[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496982911 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003325] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008538] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010008] pid_max: default: 32768 minimum: 301 [ 0.011142] LSM: Security Framework initializing [ 0.012033] Yama: becoming mindful. [ 0.013024] SELinux: Initializing. [ 0.014050] *** VALIDATE selinux *** [ 0.023060] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029009] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032086] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034087] *** VALIDATE tmpfs *** [ 0.035432] *** VALIDATE proc *** [ 0.036205] *** VALIDATE cgroup *** [ 0.037005] *** VALIDATE cgroup2 *** [ 0.039193] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041113] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043039] Spectre V2 : User space: Vulnerable [ 0.044005] Speculative Store Bypass: Vulnerable [ 0.047182] debug: unmapping init [mem 0xffffffff84659000-0xffffffff84660fff] [ 0.050000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050599] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051104] ... version: 2 [ 0.052009] ... bit width: 48 [ 0.053007] ... generic registers: 4 [ 0.054010] ... value mask: 0000ffffffffffff [ 0.055006] ... max period: 00007fffffffffff [ 0.056013] ... fixed-purpose events: 3 [ 0.057006] ... event mask: 000000070000000f [ 0.058269] rcu: Hierarchical SRCU implementation. [ 0.060336] smp: Bringing up secondary CPUs ... [ 0.061506] x86: Booting SMP configuration: [ 0.062017] .... node #0, CPUs: #1 #2 #3 [ 0.082207] smp: Brought up 1 node, 4 CPUs [ 0.084009] smpboot: Max logical packages: 1 [ 0.085008] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.210384] node 0 deferred pages initialised in 122ms [ 0.215080] devtmpfs: initialized [ 0.216261] x86/mm: Memory block size: 128MB [ 0.220554] gcov: version magic: 0x41383552 [ 0.223289] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.227071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.229299] pinctrl core: initialized pinctrl subsystem [ 0.231160] [ 0.231709] ************************************************************* [ 0.234011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.236012] ** ** [ 0.239013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.241010] ** ** [ 0.244010] ** This means that this kernel is built to expose internal ** [ 0.247015] ** IOMMU data structures, which may compromise security on ** [ 0.250012] ** your system. ** [ 0.252010] ** ** [ 0.254010] ** If you see this message and you are not debugging the ** [ 0.256010] ** kernel, report this immediately to your vendor! ** [ 0.258009] ** ** [ 0.260009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.264019] ************************************************************* [ 0.267782] NET: Registered protocol family 16 [ 0.269417] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.271118] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.274045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.278089] cpuidle: using governor menu [ 0.282816] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.285494] PCI: Using configuration type 1 for base access [ 0.287117] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.296171] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.297019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.299025] cryptd: max_cpu_qlen set to 1000 [ 0.301212] ACPI: Added _OSI(Module Device) [ 0.304047] ACPI: Added _OSI(Processor Device) [ 0.307014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.309010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.314099] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.320610] ACPI: Interpreter enabled [ 0.321045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.322009] ACPI: Using IOAPIC for interrupt routing [ 0.324082] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.327387] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.337000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.338025] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.342019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.347076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.352361] acpiphp: Slot [2] registered [ 0.354078] acpiphp: Slot [5] registered [ 0.355076] acpiphp: Slot [6] registered [ 0.357086] acpiphp: Slot [7] registered [ 0.359087] acpiphp: Slot [8] registered [ 0.361077] acpiphp: Slot [9] registered [ 0.363116] acpiphp: Slot [10] registered [ 0.365088] acpiphp: Slot [3] registered [ 0.367216] acpiphp: Slot [4] registered [ 0.368086] acpiphp: Slot [11] registered [ 0.370081] acpiphp: Slot [12] registered [ 0.372077] acpiphp: Slot [13] registered [ 0.373091] acpiphp: Slot [14] registered [ 0.375260] acpiphp: Slot [15] registered [ 0.378115] acpiphp: Slot [16] registered [ 0.380090] acpiphp: Slot [17] registered [ 0.381085] acpiphp: Slot [18] registered [ 0.383081] acpiphp: Slot [19] registered [ 0.384105] acpiphp: Slot [20] registered [ 0.386076] acpiphp: Slot [21] registered [ 0.388079] acpiphp: Slot [22] registered [ 0.389072] acpiphp: Slot [23] registered [ 0.391093] acpiphp: Slot [24] registered [ 0.392095] acpiphp: Slot [25] registered [ 0.393081] acpiphp: Slot [26] registered [ 0.394102] acpiphp: Slot [27] registered [ 0.395459] acpiphp: Slot [28] registered [ 0.397082] acpiphp: Slot [29] registered [ 0.399088] acpiphp: Slot [30] registered [ 0.400080] acpiphp: Slot [31] registered [ 0.402085] PCI host bridge to bus 0000:00 [ 0.403015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.405022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.407020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.410022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.412022] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.415028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.417209] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.421009] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.425267] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.433936] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.439057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.443022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.446014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.448012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.453516] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.458695] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.460045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.464088] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.476033] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.520028] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.533067] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.551605] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.569016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.587108] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.618026] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.627000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.638019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.645017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.675020] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.691593] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.699018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.703017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.716017] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.723590] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.728022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.735028] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.746035] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.757101] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.762021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.769019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.782021] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.791848] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.799019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.810021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.831018] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.847031] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.849411] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.851329] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.852324] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.854175] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.860128] iommu: Default domain type: Passthrough [ 0.861487] SCSI subsystem initialized [ 0.862000] ACPI: bus type USB registered [ 0.863114] usbcore: registered new interface driver usbfs [ 0.865091] usbcore: registered new interface driver hub [ 0.868179] usbcore: registered new device driver usb [ 0.870237] pps_core: LinuxPPS API ver. 1 registered [ 0.872012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.875072] PTP clock support registered [ 0.878690] EDAC MC: Ver: 3.0.0 [ 0.879716] PCI: Using ACPI for IRQ routing [ 0.882140] NetLabel: Initializing [ 0.884010] NetLabel: domain hash size = 128 [ 0.885007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.886094] NetLabel: unlabeled traffic allowed by default [ 0.889125] vgaarb: loaded [ 0.890267] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.892017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.897000] clocksource: Switched to clocksource kvm-clock [ 1.011357] VFS: Disk quotas dquot_6.6.0 [ 1.012604] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.015212] *** VALIDATE ramfs *** [ 1.016371] *** VALIDATE hugetlbfs *** [ 1.018043] pnp: PnP ACPI init [ 1.020820] pnp: PnP ACPI: found 6 devices [ 1.050190] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.053549] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.054964] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.056140] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.057475] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.059374] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 1.062111] NET: Registered protocol family 2 [ 1.064431] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.069599] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.073547] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.080043] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.083731] TCP: Hash tables configured (established 65536 bind 65536) [ 1.086602] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.088920] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.090515] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.092592] NET: Registered protocol family 1 [ 1.095394] RPC: Registered named UNIX socket transport module. [ 1.097028] RPC: Registered udp transport module. [ 1.098279] RPC: Registered tcp transport module. [ 1.099815] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.101458] NET: Registered protocol family 44 [ 1.102898] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.104561] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.106859] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.109093] PCI: CLS 0 bytes, default 64 [ 1.111100] Unpacking initramfs... [ 3.198724] debug: unmapping init [mem 0xffff8d827cc54000-0xffff8d827ffbffff] [ 3.204855] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.209183] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.212850] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.784236] Initialise system trusted keyrings [ 3.786167] Key type blacklist registered [ 3.788707] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.798084] zbud: loaded [ 3.801378] *** VALIDATE nfs *** [ 3.802701] *** VALIDATE nfs4 *** [ 3.803987] pstore: using deflate compression [ 3.807993] Platform Keyring initialized [ 4.195937] hrtimer: interrupt took 23935305 ns [ 4.880225] NET: Registered protocol family 38 [ 4.882663] Key type asymmetric registered [ 4.884625] Asymmetric key parser 'x509' registered [ 4.890132] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.918414] io scheduler mq-deadline registered [ 4.919322] io scheduler kyber registered [ 4.919322] io scheduler bfq registered [ 4.925642] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.929386] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.931904] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.935161] ACPI: Power Button [PWRF] [ 5.085343] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.287293] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.977382] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.111183] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.716361] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.783371] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.829345] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.837434] Non-volatile memory driver v1.3 [ 6.838726] Linux agpgart interface v0.103 [ 6.892291] virtio_blk virtio1: [vda] 133240 512-byte logical blocks (68.2 MB/65.1 MiB) [ 6.895607] vda: detected capacity change from 0 to 68218880 [ 7.006104] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.012172] vdb: detected capacity change from 0 to 1073741824 [ 7.431806] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.434940] vdc: detected capacity change from 0 to 2621440000 [ 7.468901] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.471975] vdd: detected capacity change from 0 to 2621440000 [ 7.599752] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.606276] vde: detected capacity change from 0 to 4294967296 [ 7.664395] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.667430] vdf: detected capacity change from 0 to 4294967296 [ 7.683845] libphy: Fixed MDIO Bus: probed [ 7.700314] usbcore: registered new interface driver usbserial_generic [ 7.702473] usbserial: USB Serial support registered for generic [ 7.706919] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.710371] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.711754] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.716900] mousedev: PS/2 mouse device common for all mice [ 7.719686] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.720179] rtc_cmos 00:05: RTC can wake from S4 [ 7.739067] rtc_cmos 00:05: registered as rtc0 [ 7.740334] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.742358] intel_pstate: CPU model not supported [ 7.750165] hid: raw HID events driver (C) Jiri Kosina [ 7.751787] usbcore: registered new interface driver usbhid [ 7.753359] usbhid: USB HID core driver [ 7.758097] drop_monitor: Initializing network drop monitor service [ 7.761080] Initializing XFRM netlink socket [ 7.773169] NET: Registered protocol family 10 [ 7.782630] Segment Routing with IPv6 [ 7.783906] NET: Registered protocol family 17 [ 7.789065] mpls_gso: MPLS GSO support [ 7.816885] RAS: Correctable Errors collector initialized. [ 7.818641] AVX version of gcm_enc/dec engaged. [ 7.819935] AES CTR mode by8 optimization enabled [ 7.836551] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.851846] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.682291] sched_clock: Marking stable (8682269507, 0)->(9746285344, -1064015837) [ 8.690231] registered taskstats version 1 [ 8.693585] Loading compiled-in X.509 certificates [ 8.699304] zswap: loaded using pool lzo/zbud [ 8.765297] Key type big_key registered [ 8.800272] Key type encrypted registered [ 8.806238] ima: No TPM chip found, activating TPM-bypass! [ 8.808235] ima: Allocated hash algorithm: sha1 [ 8.809679] ima: No architecture policies found [ 8.811578] evm: Initialising EVM extended attributes: [ 8.817146] evm: security.selinux [ 8.818502] evm: security.ima [ 8.819521] evm: security.capability [ 8.823861] evm: HMAC attrs: 0x1 [ 8.831823] rtc_cmos 00:05: setting system clock to 2025-09-25 18:07:06 UTC (1758823626) [ 8.842602] debug: unmapping init [mem 0xffffffff85603000-0xffffffff857fffff] [ 8.849708] debug: unmapping init [mem 0xffffffff84382000-0xffffffff84658fff] [ 8.860048] Write protecting the kernel read-only data: 28672k [ 8.863445] debug: unmapping init [mem 0xffffffff82a03000-0xffffffff82bfffff] [ 8.870409] debug: unmapping init [mem 0xffffffff83314000-0xffffffff833fffff] [ 8.956609] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.972614] systemd[1]: Detected virtualization kvm. [ 8.974680] systemd[1]: Detected architecture x86-64. [ 8.980185] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 9.019232] systemd[1]: No hostname configured. [ 9.025805] systemd[1]: Set hostname to . [ 9.028210] random: systemd: uninitialized urandom read (16 bytes read) [ 9.030711] systemd[1]: Initializing machine ID from random generator. [ 9.359854] random: systemd: uninitialized urandom read (16 bytes read) [ 9.363967] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 9.371365] random: systemd: uninitialized urandom read (16 bytes read) [ 9.376185] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 9.384859] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.712643] device-mapper: uevent: version 1.0.3 [ 10.714851] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.185843] virtio_net virtio0 ens2: renamed from eth0 [ 12.363268] random: fast init done [ 12.814093] scsi host0: ata_piix [ 12.836272] scsi host1: ata_piix [ 12.837713] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.842565] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.335136] random: crng init done [ 17.348093] random: 7 urandom warning(s) missed due to ratelimiting [ 17.355084] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.553229] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.837888] printk: systemd: 19 output lines suppressed due to ratelimiting [ 22.693577] SELinux: Disabled at runtime. [ 22.849590] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.875826] systemd[1]: Detected virtualization kvm. [ 22.878907] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.199216] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.202795] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.209274] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.214335] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.217413] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.239241] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.253225] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket[ 24.345612] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS . [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 25.149899] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.908642] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.909927] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.400926] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.485181] EDAC sbridge: Ver: 1.1.2 [ 30.921031] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 32.883777] NFS: Registering the id_resolver key type [ 32.885843] Key type id_resolver registered [ 32.887594] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Crash recovery kernel arming. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg124-server login: [ 56.090900] libcfs: loading out-of-tree module taints kernel. [ 56.099560] Key type ._llcrypt registered [ 56.100727] Key type .llcrypt registered [ 56.142361] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 63.576033] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 64.287737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.295224] alg: No test for adler32 (adler32-zlib) [ 65.315186] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 65.666613] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 67.288250] Key type lgssc registered [ 67.930132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 74.173113] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 75.418511] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.161314] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 81.749080] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.338063] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 89.421835] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 93.788118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.811161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 93.821357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 94.920654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 94.932956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 94.977037] Lustre: lustre-MDT0000: new disk, initializing [ 95.004110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 95.013249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 96.308885] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.560028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.593038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 101.624669] Lustre: 6484:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 101.638297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 101.639912] Lustre: Skipped 1 previous similar message [ 101.682693] Lustre: lustre-MDT0001: new disk, initializing [ 101.708356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 101.717688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 101.721065] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 103.172371] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 105.503950] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 108.828621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.855633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 108.956730] Lustre: lustre-OST0000: new disk, initializing [ 108.959545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 108.987357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 111.013904] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 115.187610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 115.191093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 115.223706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 116.940937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 116.972412] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 117.009986] Lustre: lustre-OST0001: new disk, initializing [ 117.011596] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 117.028248] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 118.990700] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 122.347989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 122.351063] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 122.361113] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 124.814207] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 129.442676] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 136.398918] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing check_logdir /tmp/testlogs/ [ 137.717144] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing yml_node [ 139.063493] Lustre: DEBUG MARKER: Client: 2.16.58.75 [ 139.861566] Lustre: DEBUG MARKER: MDS: 2.16.58.75 [ 140.603606] Lustre: DEBUG MARKER: OSS: 2.16.58.75 [ 141.127352] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 25 14:09:18 EDT 2025 [ 146.844826] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 147.396861] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 163.296719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 163.297590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 163.297780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 163.302136] Lustre: Skipped 3 previous similar messages [ 165.308351] LustreError: 12188:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 165.338095] Lustre: server umount lustre-MDT0000 complete [ 166.693938] LustreError: 10254:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758823784 with bad export cookie 7334177294122884195 [ 166.695917] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 166.697335] LustreError: 10254:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 168.416832] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 168.417164] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 168.421291] Lustre: Skipped 1 previous similar message [ 168.424313] Lustre: Skipped 3 previous similar messages [ 171.424741] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 171.427636] Lustre: Skipped 2 previous similar messages [ 172.862550] LustreError: 12390:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 172.865676] LustreError: 12390:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 172.960498] Lustre: server umount lustre-MDT0001 complete [ 174.365199] LustreError: 12592:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 174.367215] LustreError: 12592:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 174.388678] Lustre: server umount lustre-OST0000 complete [ 175.863718] Lustre: server umount lustre-OST0001 complete [ 178.225568] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 181.713182] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 185.605865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 187.993953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 190.237253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 192.442563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 194.952693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 194.982234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.077140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 195.091564] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 195.138395] Lustre: lustre-MDT0000: new disk, initializing [ 195.167580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 195.175619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 196.679537] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 201.056288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 201.087610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 201.126766] Lustre: 14969:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 201.257303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 202.731144] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 206.066696] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 207.361448] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 209.697193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 209.728242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 209.837536] Lustre: lustre-OST0000: new disk, initializing [ 209.839014] Lustre: Skipped 1 previous similar message [ 209.840802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 209.842620] Lustre: Skipped 2 previous similar messages [ 209.872623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.874500] Lustre: Skipped 1 previous similar message [ 211.199427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 211.201934] Lustre: Skipped 1 previous similar message [ 211.204063] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 211.241103] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 211.947950] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 215.113205] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 216.586926] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 216.676342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 218.063591] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 218.166883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 220.846127] LustreError: 17909:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 220.847868] LustreError: 17909:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 220.908412] Lustre: server umount lustre-OST0000 complete [ 226.787403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 226.787714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 226.788554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 226.795921] Lustre: Skipped 1 previous similar message [ 231.905342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 231.907164] Lustre: Skipped 2 previous similar messages [ 231.909311] LustreError: 18109:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 231.909317] LustreError: 18109:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 231.930324] Lustre: server umount lustre-MDT0000 complete [ 233.189863] LustreError: 14960:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758823850 with bad export cookie 7334177294122885735 [ 233.193102] LustreError: 14960:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 233.194386] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 242.669137] LustreError: 18312:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 242.672872] LustreError: 18312:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 242.852803] Lustre: server umount lustre-MDT0001 complete [ 245.808540] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 14:11:02 (1758823862) [ 247.578074] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 248.643767] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 14:11:05 (1758823865) [ 252.831744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 253.092125] LustreError: 18984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 253.105591] LustreError: 18984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 253.150509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 256.018712] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 258.529459] LustreError: 18984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 261.708261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 264.559949] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 268.584226] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 271.301178] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 276.810707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 283.880421] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 289.331329] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 296.929607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.931696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 296.952038] Lustre: Skipped 1 previous similar message [ 302.776385] LustreError: 21226:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 302.782674] LustreError: 21226:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 302.908400] Lustre: server umount lustre-OST0000 complete [ 307.169849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.179467] Lustre: Skipped 1 previous similar message [ 311.430000] Lustre: server umount lustre-MDT0000 complete [ 312.289672] LustreError: 18979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.295547] LustreError: 18979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 313.263580] LustreError: 18965:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758823930 with bad export cookie 7334177294122886617 [ 313.267156] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 313.270118] LustreError: 18965:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 313.517202] Lustre: server umount lustre-MDT0001 complete [ 318.882128] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 320.422417] Key type lgssc unregistered [ 320.638412] LNet: 22164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 320.642255] LNetError: 22164:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 320.659534] LNet: Removed LNI 192.168.201.124@tcp [ 321.200163] Key type .llcrypt unregistered [ 321.201657] Key type ._llcrypt unregistered [ 332.403755] Key type ._llcrypt registered [ 332.407389] Key type .llcrypt registered [ 332.454329] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 332.824408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 332.831927] alg: No test for adler32 (adler32-zlib) [ 333.725297] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 333.830376] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 335.432183] Key type lgssc registered [ 335.890695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 339.946856] loop: module loaded [ 339.974427] loop0: detected capacity change from 0 to 204800000 [ 340.055232] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 340.063294] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 340.245177] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 340.250268] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 340.278158] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 340.306557] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 340.328462] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 340.363124] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 343.141262] loop0: detected capacity change from 0 to 204800000 [ 343.274339] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 343.280184] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 343.286452] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 343.347755] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 345.623866] loop0: detected capacity change from 0 to 204800000 [ 348.389488] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 348.429227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 348.439934] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 349.583067] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 349.608595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 349.668602] Lustre: lustre-MDT0000: new disk, initializing [ 349.733172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 349.743571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 351.677829] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 356.667926] loop1: detected capacity change from 0 to 204800000 [ 359.509622] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 359.546588] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 359.699938] Lustre: lustre-OST07c6: new disk, initializing [ 359.703578] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 359.707961] Lustre: Skipped 1 previous similar message [ 359.744920] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 362.578294] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 368.647397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 368.650323] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 368.665626] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 449.136105] LustreError: 26492:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 449.227682] Lustre: server umount lustre-OST07c6 complete [ 463.849467] LustreError: 26991:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 463.852442] LustreError: 26991:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 463.967476] Lustre: server umount lustre-MDT0000 complete [ 472.271142] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 474.038568] Key type lgssc unregistered [ 474.256353] LNet: 27825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.262151] LNetError: 27825:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 474.275570] LNet: Removed LNI 192.168.201.124@tcp [ 474.782649] Key type .llcrypt unregistered [ 474.784626] Key type ._llcrypt unregistered [ 484.437126] Key type ._llcrypt registered [ 484.440416] Key type .llcrypt registered [ 484.504685] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 484.928515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 484.947884] alg: No test for adler32 (adler32-zlib) [ 485.907861] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 486.100968] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 487.752472] Key type lgssc registered [ 488.399146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 493.154153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 493.166609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.493539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.341483] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 501.437169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 501.710970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 504.131845] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 507.678875] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 509.691993] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 513.142218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 513.380071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 516.045446] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 518.552636] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 519.530145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 531.064075] LustreError: 31274:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 537.057201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 537.060735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.062243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 537.069071] Lustre: Skipped 1 previous similar message [ 541.108227] LustreError: 31523:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 541.238332] Lustre: server umount lustre-OST0000 complete [ 547.299672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.300328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 547.305363] Lustre: Skipped 1 previous similar message [ 547.307825] Lustre: Skipped 1 previous similar message [ 549.314283] LustreError: 31725:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 549.316695] LustreError: 31725:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 549.365264] Lustre: server umount lustre-MDT0000 complete [ 551.259950] LustreError: 29035:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824168 with bad export cookie 1031842941583937431 [ 551.261239] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 551.269274] LustreError: 29035:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 551.349852] LustreError: 31927:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 551.351729] LustreError: 31927:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 551.470655] Lustre: server umount lustre-MDT0001 complete [ 556.742262] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 558.196117] Key type lgssc unregistered [ 558.375602] LNet: 32459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 558.381839] LNetError: 32459:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 558.390181] LNet: Removed LNI 192.168.201.124@tcp [ 558.791843] Key type .llcrypt unregistered [ 558.794425] Key type ._llcrypt unregistered [ 569.486208] Key type ._llcrypt registered [ 569.487968] Key type .llcrypt registered [ 569.559371] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 570.139911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 570.159789] alg: No test for adler32 (adler32-zlib) [ 571.095885] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 571.229499] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 572.856159] Key type lgssc registered [ 573.360611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 577.353907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 577.368529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.608099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 580.606714] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 585.703831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 585.932906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 588.016142] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 591.006425] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 592.959813] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 596.734262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 596.978070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 600.387148] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 603.494674] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 604.145673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 619.491511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 619.495572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.507582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 621.538196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 621.538340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 621.541989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 624.754389] LustreError: 35953:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 624.862570] Lustre: server umount lustre-OST0000 complete [ 627.169270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 627.172825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.181585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 627.187832] Lustre: Skipped 1 previous similar message [ 631.780159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 631.780376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 631.788708] Lustre: Skipped 1 previous similar message [ 632.848378] LustreError: 36154:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 632.850499] LustreError: 36154:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 632.891059] Lustre: server umount lustre-MDT0000 complete [ 634.830093] LustreError: 36155:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824252 with bad export cookie 12822198545887573304 [ 634.833968] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 634.836242] LustreError: 36155:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 634.921699] LustreError: 36357:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 634.924681] LustreError: 36357:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 635.055108] Lustre: server umount lustre-MDT0001 complete [ 640.322080] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 641.530479] Key type lgssc unregistered [ 641.672534] LNet: 36888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.676659] LNetError: 36888:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 641.685425] LNet: Removed LNI 192.168.201.124@tcp [ 642.018753] Key type .llcrypt unregistered [ 642.020264] Key type ._llcrypt unregistered [ 653.887085] Key type ._llcrypt registered [ 653.888469] Key type .llcrypt registered [ 653.937632] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 654.288112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 654.329440] alg: No test for adler32 (adler32-zlib) [ 655.217242] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 655.312694] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 656.904518] Key type lgssc registered [ 657.322642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 660.295352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 660.305928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 661.535660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 663.194570] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 666.836552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 667.017447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 668.693925] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 671.114541] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 672.781845] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 675.669767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 675.830781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 677.927506] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 680.053673] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 687.085176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 696.962080] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.24@tcp (stopping) [ 697.316523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 697.321951] Lustre: Skipped 2 previous similar messages [ 700.613221] LustreError: 40529:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 700.644939] Lustre: server umount lustre-MDT0000 complete [ 702.070486] LustreError: 38257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 702.077651] LustreError: 38257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 702.154614] LustreError: 38243:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824319 with bad export cookie 3491418764000827983 [ 702.158393] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 702.159873] LustreError: 38243:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 702.433937] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.438602] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 702.441446] Lustre: Skipped 3 previous similar messages [ 707.189626] LustreError: 40421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 707.197197] LustreError: 40421:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 707.200173] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.24@tcp (stopping) [ 708.299719] LustreError: 40732:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 708.301826] LustreError: 40732:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 708.396941] Lustre: server umount lustre-MDT0001 complete [ 726.571888] LustreError: 40955:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 726.574620] LustreError: 40955:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 726.645203] Lustre: server umount lustre-OST0000 complete [ 733.358214] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 734.719807] Key type lgssc unregistered [ 734.853580] LNet: 41791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.856787] LNetError: 41791:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.866370] LNet: Removed LNI 192.168.201.124@tcp [ 735.223285] Key type .llcrypt unregistered [ 735.224810] Key type ._llcrypt unregistered [ 746.082284] Key type ._llcrypt registered [ 746.084202] Key type .llcrypt registered [ 746.125823] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 746.513480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 746.548731] alg: No test for adler32 (adler32-zlib) [ 747.420828] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 747.538566] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 749.144184] Key type lgssc registered [ 749.632417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 753.127409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 753.147894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 754.369624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 755.903193] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 759.306865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 759.469348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 761.069132] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 763.347338] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 764.935550] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 767.766476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 767.920562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 770.391669] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 772.825699] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 780.203192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 788.609535] Lustre: lustre-OST0000: Not available for connect from 192.168.201.24@tcp (stopping) [ 790.500542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 790.501616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 790.503984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 790.507710] Lustre: Skipped 1 previous similar message [ 793.718733] Lustre: lustre-OST0000: Not available for connect from 192.168.201.24@tcp (stopping) [ 793.722293] Lustre: Skipped 1 previous similar message [ 798.837672] Lustre: lustre-OST0000: Not available for connect from 192.168.201.24@tcp (stopping) [ 798.841240] Lustre: Skipped 2 previous similar messages [ 800.224159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 800.249082] LustreError: 45450:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 800.321224] Lustre: server umount lustre-OST0000 complete [ 813.511869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 813.512715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 813.516572] Lustre: Skipped 1 previous similar message [ 819.713406] LustreError: 45828:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 819.716112] LustreError: 45828:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 819.751411] Lustre: server umount lustre-MDT0000 complete [ 820.706479] LustreError: 43183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 820.714798] LustreError: 43183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 821.294750] LustreError: 43162:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824438 with bad export cookie 17012712538661063012 [ 821.299297] LustreError: 43162:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 821.300314] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 827.464163] LustreError: 46030:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 827.466369] LustreError: 46030:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 827.571720] Lustre: server umount lustre-MDT0001 complete [ 832.111984] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 833.321678] Key type lgssc unregistered [ 833.472418] LNet: 46562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.475852] LNetError: 46562:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 833.484358] LNet: Removed LNI 192.168.201.124@tcp [ 833.830741] Key type .llcrypt unregistered [ 833.832954] Key type ._llcrypt unregistered [ 844.227287] Key type ._llcrypt registered [ 844.228684] Key type .llcrypt registered [ 844.267732] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 844.599261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 844.607808] alg: No test for adler32 (adler32-zlib) [ 845.485087] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 845.576901] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 847.168336] Key type lgssc registered [ 847.613087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 850.671606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 850.681806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.911737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 853.439602] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 856.754494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 856.920678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 858.447730] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 860.455374] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 861.823392] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 864.448191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 864.583971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.817813] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 869.002647] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 876.902913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 882.088706] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 883.556413] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 894.461078] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-09-2514h21m51s [ 895.066184] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 897.506568] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.507923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 897.510912] Lustre: Skipped 2 previous similar messages [ 897.512871] Lustre: Skipped 1 previous similar message [ 898.530040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 898.532736] Lustre: Skipped 3 previous similar messages [ 900.222115] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.24@tcp (stopping) [ 904.674410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 904.677009] Lustre: Skipped 3 previous similar messages [ 909.795496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 909.798347] Lustre: Skipped 1 previous similar message [ 910.304159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 910.337893] LustreError: 50520:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 910.373079] Lustre: server umount lustre-MDT0000 complete [ 910.819167] LustreError: 47936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 910.825571] LustreError: 47936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 911.803543] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 912.500810] LustreError: 50318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 914.529091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 914.580487] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 914.690213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 914.731369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 915.751568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 916.237323] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 917.618802] Lustre: DEBUG MARKER: Wait for df (35953) ... [ 918.284303] Lustre: DEBUG MARKER: done [ 920.034428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 920.038216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 920.044063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 921.059080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 921.648282] LustreError: 51573:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 921.650540] LustreError: 51573:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 921.724744] Lustre: server umount lustre-OST0000 complete [ 925.452912] LustreError: 51776:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 925.456054] LustreError: 51776:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 925.492978] Lustre: server umount lustre-MDT0000 complete [ 926.178090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 926.178095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 926.187506] LustreError: 47933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.195471] LustreError: 47933:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 927.080637] LustreError: 47917:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824544 with bad export cookie 14177804796604685380 [ 927.083837] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 927.085271] LustreError: 47917:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 929.154198] LustreError: 51977:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 929.157538] LustreError: 51977:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 929.266344] Lustre: server umount lustre-MDT0001 complete [ 933.693770] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 934.962407] Key type lgssc unregistered [ 935.115587] LNet: 52509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.118889] LNetError: 52509:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.129246] LNet: Removed LNI 192.168.201.124@tcp [ 935.500937] Key type .llcrypt unregistered [ 935.503677] Key type ._llcrypt unregistered [ 948.318164] Key type ._llcrypt registered [ 948.319878] Key type .llcrypt registered [ 948.364832] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 955.661331] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 956.074086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 956.100863] alg: No test for adler32 (adler32-zlib) [ 956.979068] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 957.078862] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 958.664155] Key type lgssc registered [ 959.116658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 962.701490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 965.287896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 967.722995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 970.466586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 973.312382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 973.335042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 973.344592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.443199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 974.465111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 974.506441] Lustre: lustre-MDT0000: new disk, initializing [ 974.546095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 974.555509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 976.047460] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 980.572466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 980.605286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 980.644162] Lustre: 56433:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 980.671289] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 980.673968] Lustre: Skipped 1 previous similar message [ 980.713749] Lustre: lustre-MDT0001: new disk, initializing [ 980.758755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 980.777264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 980.781849] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 982.330818] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 985.997705] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 987.554916] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 990.180597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 990.206232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 990.315634] Lustre: lustre-OST0000: new disk, initializing [ 990.318623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 990.353363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.379630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 990.384345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 990.439567] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 992.605464] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 996.226267] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 998.206037] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 998.316728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1000.351345] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1000.485768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1006.051841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1006.052950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1006.052986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1006.056701] Lustre: Skipped 1 previous similar message [ 1007.472394] LustreError: 59375:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1007.559540] Lustre: server umount lustre-OST0000 complete [ 1011.169230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1011.170615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1011.173395] Lustre: Skipped 1 previous similar message [ 1011.175207] Lustre: Skipped 1 previous similar message [ 1015.233938] LustreError: 59576:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1015.237812] LustreError: 59576:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1015.266141] Lustre: server umount lustre-MDT0000 complete [ 1016.758219] LustreError: 57213:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824634 with bad export cookie 6856205723671335086 [ 1016.761143] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1016.763594] LustreError: 57213:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1016.827951] LustreError: 59778:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1016.830548] LustreError: 59778:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1016.914567] Lustre: server umount lustre-MDT0001 complete [ 1020.470769] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 14:23:57 (1758824637) [ 1023.456527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.691903] LustreError: 60448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1023.700952] LustreError: 60448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1023.726731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1025.438186] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1029.059118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1029.089781] LustreError: 60448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1029.242600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1030.784543] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1032.878717] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1034.237638] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1036.833084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1039.079778] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1041.183855] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1054.359304] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-09-2514h24m31s [ 1054.964464] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1057.811320] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1058.784072] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1058.803911] LustreError: 60443:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1078.917124] Lustre: lustre-MDT0000: Client 6d54a6cb-b741-4cc5-a5e6-93b0aeeb12ff (at 192.168.201.24@tcp) reconnecting [ 1098.848129] LustreError: 60443:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1099.738639] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1100.761437] Lustre: DEBUG MARKER: done [ 1104.866559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1104.869306] LustreError: Skipped 1 previous similar message [ 1104.871370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1104.877708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1104.879779] Lustre: Skipped 1 previous similar message [ 1105.889233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1106.915024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1106.917808] Lustre: Skipped 2 previous similar messages [ 1108.782079] LustreError: 63472:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1108.785016] LustreError: 63472:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1108.856789] Lustre: server umount lustre-OST0000 complete [ 1111.008972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1111.012602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1111.016757] Lustre: Skipped 1 previous similar message [ 1111.018848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1116.421268] LustreError: 63673:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1116.424574] LustreError: 63673:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1116.474261] Lustre: server umount lustre-MDT0000 complete [ 1117.090115] LustreError: 60448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1117.096215] LustreError: 60448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1118.131289] LustreError: 60427:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824735 with bad export cookie 6856205723671335954 [ 1118.134915] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1118.138755] LustreError: 60427:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1118.306862] Lustre: server umount lustre-MDT0001 complete [ 1122.597549] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1123.721827] Key type lgssc unregistered [ 1123.853599] LNet: 64405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1123.856739] LNetError: 64405:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1123.865550] LNet: Removed LNI 192.168.201.124@tcp [ 1124.218913] Key type .llcrypt unregistered [ 1124.220641] Key type ._llcrypt unregistered [ 1136.567029] Key type ._llcrypt registered [ 1136.568564] Key type .llcrypt registered [ 1136.614788] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 1144.211824] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1144.695727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1144.707523] alg: No test for adler32 (adler32-zlib) [ 1145.581705] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1145.685773] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1147.272159] Key type lgssc registered [ 1147.688316] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1151.400767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1154.283843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1157.064894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1160.036092] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1162.996213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1163.024921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1163.037294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1164.149959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1164.167705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1164.208862] Lustre: lustre-MDT0000: new disk, initializing [ 1164.250937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1164.261449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1165.957377] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1170.876866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1170.905539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1170.947612] Lustre: 68330:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1170.977759] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1170.980466] Lustre: Skipped 1 previous similar message [ 1171.019495] Lustre: lustre-MDT0001: new disk, initializing [ 1171.063854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1171.084828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1171.089800] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1172.676099] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1176.479954] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1178.052272] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1180.831727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1180.862678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1180.982596] Lustre: lustre-OST0000: new disk, initializing [ 1180.985376] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1181.027764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1183.275960] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1186.936863] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1188.349750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1188.353087] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1188.399054] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1188.564475] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1188.649918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1190.186506] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1190.273264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1193.442482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1193.443125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.446140] LustreError: Skipped 1 previous similar message [ 1193.452881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1193.456859] Lustre: Skipped 1 previous similar message [ 1197.044178] LustreError: 71269:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1197.110840] Lustre: server umount lustre-OST0000 complete [ 1202.145279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1202.149274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.154488] Lustre: Skipped 1 previous similar message [ 1202.157046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1203.681354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1203.681778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.673139] LustreError: 71472:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1204.675977] LustreError: 71472:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1204.707101] Lustre: server umount lustre-MDT0000 complete [ 1206.204601] LustreError: 70075:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824823 with bad export cookie 14993348369054445065 [ 1206.207941] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1206.209113] LustreError: 70075:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1206.278743] LustreError: 71674:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1206.282747] LustreError: 71674:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1206.386794] Lustre: server umount lustre-MDT0001 complete [ 1209.750285] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 14:27:07 (1758824827) [ 1210.347878] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1211.047703] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 14:27:08 (1758824828) [ 1216.014228] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1217.619285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.797084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1218.186450] Lustre: Failing over lustre-MDT0000 [ 1218.189073] LustreError: 72749:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1218.194804] LustreError: 72749:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1218.199456] LustreError: 72749:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1218.265956] LustreError: 72800:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1218.269148] LustreError: 72800:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1218.353957] Lustre: server umount lustre-MDT0000 complete [ 1220.909408] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 14:27:18 (1758824838) [ 1223.327852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.473618] LustreError: 73368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1223.481958] LustreError: 73368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1223.490419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1224.933620] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1228.300618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1229.966420] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1232.097406] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1233.626269] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1236.456711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1236.598391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1236.601239] Lustre: Skipped 1 previous similar message [ 1238.938298] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1241.218615] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1254.281789] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1259.490511] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1259.491798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1259.492652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1259.492658] Lustre: Skipped 1 previous similar message [ 1259.495206] Lustre: Skipped 2 previous similar messages [ 1262.466244] LustreError: 75736:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1262.468598] LustreError: 75736:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1262.497034] Lustre: server umount lustre-MDT0000 complete [ 1263.985908] LustreError: 74987:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824881 with bad export cookie 14993348369054446206 [ 1263.988858] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1263.991157] LustreError: 74987:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1264.163686] Lustre: server umount lustre-MDT0001 complete [ 1265.599273] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1267.529982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1270.949970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.760691] LustreError: 66542:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d2619172 x1844261434746880/t0(0) o250->MGC192.168.201.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1273.889656] LustreError: 74975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.923750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1273.956586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1275.448689] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1278.848741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1278.946242] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1278.947051] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1278.949413] Lustre: Skipped 2 previous similar messages [ 1280.043085] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1280.605244] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1281.056334] Lustre: 66543:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1758824882/real 1758824882] req@00000000bf5411d0 x1844261434746496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1758824898 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1282.802090] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1284.219249] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1286.112164] Lustre: 66544:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1758824886/real 1758824886] req@00000000fd3345b2 x1844261434746624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1758824902 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1286.122211] Lustre: 66544:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1289.185232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1289.188345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.193591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1289.440152] Lustre: 66546:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1758824891/real 1758824891] req@0000000099b00721 x1844261434747008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1758824907 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1292.481748] LustreError: 78197:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1292.484661] LustreError: 78197:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1292.516935] Lustre: server umount lustre-MDT0000 complete [ 1294.069715] LustreError: 73348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824911 with bad export cookie 14993348369054448096 [ 1294.074844] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1294.075411] LustreError: 73348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1294.233678] Lustre: server umount lustre-MDT0001 complete [ 1296.809500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1300.588969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1303.456642] LustreError: 66542:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d86f4fda x1844261434767488/t0(0) o250->MGC192.168.201.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1303.586965] LustreError: 74974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.649955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1305.173480] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1308.605979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1308.642516] LustreError: 74974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1308.642754] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1308.659353] Lustre: Skipped 2 previous similar messages [ 1308.765950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1308.768702] Lustre: Skipped 2 previous similar messages [ 1309.797966] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1309.801678] Lustre: Skipped 1 previous similar message [ 1310.450809] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1310.817097] Lustre: 66543:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1758824912/real 1758824912] req@0000000019b54c0a x1844261434767104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1758824928 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1312.736441] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1314.208570] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1317.152147] Lustre: 66545:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1758824918/real 1758824918] req@00000000b207b69c x1844261434767360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1758824934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1317.161452] Lustre: 66545:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1318.881546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1318.885677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1318.888481] Lustre: Skipped 3 previous similar messages [ 1322.237824] LustreError: 80652:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1322.240356] LustreError: 80652:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1322.266408] Lustre: server umount lustre-MDT0000 complete [ 1323.681280] LustreError: 73349:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824941 with bad export cookie 14993348369054449468 [ 1323.683830] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1323.686248] LustreError: 73349:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1323.833134] Lustre: server umount lustre-MDT0001 complete [ 1326.304441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1327.613098] Lustre: DEBUG MARKER: files compared the same [ 1337.657354] Lustre: server umount lustre-OST0000 complete [ 1343.760123] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1344.935139] Key type lgssc unregistered [ 1345.082632] LNet: 82282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.086201] LNetError: 82282:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1345.093367] LNet: Removed LNI 192.168.201.124@tcp [ 1345.448398] Key type .llcrypt unregistered [ 1345.450129] Key type ._llcrypt unregistered [ 1354.444001] Key type ._llcrypt registered [ 1354.445648] Key type .llcrypt registered [ 1354.482786] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1354.905126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1354.911319] alg: No test for adler32 (adler32-zlib) [ 1355.764394] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1355.847607] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1357.432159] Key type lgssc registered [ 1357.804838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1360.437258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1360.445534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.604858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1363.008814] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug +malloc all [ 1365.959633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1366.058355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1367.468815] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug +malloc all [ 1369.505243] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1370.871853] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1373.298937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1373.395165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.444445] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug +malloc all [ 1377.393400] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1386.724675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1391.072810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1391.076962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1391.083285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1392.097497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1392.097603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1392.097622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.109240] Lustre: Skipped 1 previous similar message [ 1397.102102] LustreError: 85930:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1397.170533] Lustre: server umount lustre-OST0000 complete [ 1401.824600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1401.827713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1401.831859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1404.799214] LustreError: 86133:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1404.801415] LustreError: 86133:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1404.826222] Lustre: server umount lustre-MDT0000 complete [ 1406.176577] LustreError: 84383:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825023 with bad export cookie 6572720219862700791 [ 1406.177622] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1406.181480] LustreError: 84383:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1406.238603] LustreError: 86336:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1406.240637] LustreError: 86336:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1406.321538] Lustre: server umount lustre-MDT0001 complete [ 1410.233665] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1411.369745] Key type lgssc unregistered [ 1411.510500] LNet: 86867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.514320] LNetError: 86867:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.527394] LNet: Removed LNI 192.168.201.124@tcp [ 1411.901370] Key type .llcrypt unregistered [ 1411.903258] Key type ._llcrypt unregistered [ 1422.364726] Key type ._llcrypt registered [ 1422.366630] Key type .llcrypt registered [ 1422.409393] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1422.805414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1422.843447] alg: No test for adler32 (adler32-zlib) [ 1423.713358] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1423.808334] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1425.400179] Key type lgssc registered [ 1425.833422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1428.738904] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1428.746986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1444.192373] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1449.312174] LustreError: 88214:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 1475.936165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1477.907324] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1479.769902] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1482.346608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1482.388617] Lustre: *** cfs_fail_loc=706, val=0*** [ 1482.516321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1482.537621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1483.895149] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1486.726416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1486.855619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1488.112400] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1489.901779] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1491.110321] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1491.937248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1491.939316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1491.942602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1492.960927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1492.964321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1492.968559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1493.805069] LustreError: 90536:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1493.864247] Lustre: server umount lustre-OST0000 complete [ 1496.034311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1496.035078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1496.039485] Lustre: Skipped 1 previous similar message [ 1501.153111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1501.154838] Lustre: Skipped 2 previous similar messages [ 1501.244329] LustreError: 90739:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1501.246153] LustreError: 90739:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1501.270412] Lustre: server umount lustre-MDT0000 complete [ 1502.516197] LustreError: 90740:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825120 with bad export cookie 14541692107427189602 [ 1502.518895] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1502.520545] LustreError: 90740:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1502.585857] LustreError: 90942:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1502.588661] LustreError: 90942:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1502.662987] Lustre: server umount lustre-MDT0001 complete [ 1506.413111] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1507.475566] Key type lgssc unregistered [ 1507.605434] LNet: 91473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1507.607860] LNetError: 91473:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1507.614389] LNet: Removed LNI 192.168.201.124@tcp [ 1507.935261] Key type .llcrypt unregistered [ 1507.936751] Key type ._llcrypt unregistered [ 1517.613906] Key type ._llcrypt registered [ 1517.615097] Key type .llcrypt registered [ 1517.650835] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1517.974739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1517.984182] alg: No test for adler32 (adler32-zlib) [ 1518.842345] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1518.934573] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1520.520139] Key type lgssc registered [ 1520.948467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1523.914040] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1523.921970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1526.236358] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1528.804244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1528.971563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1530.373972] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1533.490749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1533.605704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1535.709563] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1538.723184] LustreError: 93520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1538.730598] LustreError: 93520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 1538.859056] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1538.864323] Lustre: MGS: Client c2415773-2e91-4e5e-9cb2-61b3cc739a54 (at 0@lo) reconnecting [ 1538.946677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1538.985136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1540.348581] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1549.285387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1549.286656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1549.291180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1549.296228] Lustre: Skipped 1 previous similar message [ 1554.404034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1554.406786] Lustre: Skipped 1 previous similar message [ 1554.605184] LustreError: 95479:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1554.668742] Lustre: server umount lustre-OST0000 complete [ 1559.522288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1559.522408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1559.523032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1559.534911] Lustre: Skipped 2 previous similar messages [ 1562.237667] LustreError: 95681:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1562.239498] LustreError: 95681:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1562.248124] Lustre: server umount lustre-MDT0000 complete [ 1563.606224] LustreError: 95884:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1563.608336] LustreError: 95884:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1563.680567] Lustre: server umount lustre-MDT0001 complete [ 1565.021067] Lustre: server umount MGS complete [ 1568.802988] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:33:06 (1758825186) [ 1578.556764] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 1581.818970] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1585.380731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1587.766750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1590.119867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1592.523211] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1594.980723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1595.007824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1596.447560] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1600.657152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1600.682184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1600.767503] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1600.792088] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1600.830931] Lustre: lustre-MDT0001: new disk, initializing [ 1600.866583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1600.879948] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1600.883763] Lustre: 100595:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1600.887711] LustreError: 100595:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1602.342695] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1606.851883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1606.874468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1606.956888] Lustre: lustre-OST0000: new disk, initializing [ 1606.958668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1606.985260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1608.957968] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1611.168899] Lustre: 101790:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000007e4c03e1 x1844261825839104/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1758825244 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1612.210361] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1614.625746] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1614.631238] Lustre: MGS: Client 0e26a371-a2f6-4568-9e5d-3150cf3320a0 (at 0@lo) reconnecting [ 1614.644976] Lustre: 99879:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1614.653287] LustreError: 99879:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1614.666123] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1614.702977] Lustre: lustre-MDT0000: new disk, initializing [ 1614.766747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1615.840676] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1615.840903] Lustre: 102310:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000009b8a345d x1844261825844480/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1758825249 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1615.853670] Lustre: 102310:0:(client.c:1627:after_reply()) Skipped 1 previous similar message [ 1616.175473] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1620.259407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1620.262958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1620.899389] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1620.901963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1640.928936] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1640.931656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1640.935993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1640.939036] Lustre: Skipped 1 previous similar message [ 1645.028237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1645.030985] Lustre: Skipped 2 previous similar messages [ 1646.384145] LustreError: 102833:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1646.387366] LustreError: 102833:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1646.450587] Lustre: server umount lustre-OST0000 complete [ 1650.147100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1650.147250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1650.151902] Lustre: Skipped 2 previous similar messages [ 1650.157281] LustreError: Skipped 1 previous similar message [ 1654.015739] LustreError: 103034:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1654.018463] LustreError: 103034:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1654.027224] Lustre: server umount lustre-MDT0000 complete [ 1655.266100] LustreError: 101356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1655.271089] LustreError: 101356:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1661.578660] Lustre: server umount lustre-MDT0001 complete [ 1662.848937] LustreError: 103438:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1662.850836] LustreError: 103438:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1662.873515] Lustre: server umount MGS complete [ 1666.975970] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:34:44 (1758825284) [ 1673.066368] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1674.146253] Key type lgssc unregistered [ 1674.280322] LNet: 104772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.282325] LNetError: 104772:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.289494] LNet: Removed LNI 192.168.201.124@tcp [ 1674.593932] Key type .llcrypt unregistered [ 1674.595524] Key type ._llcrypt unregistered [ 1682.285120] Key type ._llcrypt registered [ 1682.285946] Key type .llcrypt registered [ 1682.321877] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1682.628897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1682.637695] alg: No test for adler32 (adler32-zlib) [ 1683.488237] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1683.577586] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1685.160194] Key type lgssc registered [ 1685.581930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1688.361020] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1688.369595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1689.537397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1690.858016] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1693.950453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1694.075856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1695.359033] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1697.253330] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1698.541807] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1700.782597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1700.896889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1702.943000] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1704.892240] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1715.239568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1719.264878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1719.267470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1719.271343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1720.290922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1720.291433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1720.291551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1720.299279] Lustre: Skipped 1 previous similar message [ 1723.885126] LustreError: 108217:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1723.944739] Lustre: server umount lustre-OST0000 complete [ 1725.410190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1725.410864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1725.415558] Lustre: Skipped 1 previous similar message [ 1730.529077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1730.531571] Lustre: Skipped 2 previous similar messages [ 1731.322763] LustreError: 108421:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1731.324572] LustreError: 108421:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1731.348858] Lustre: server umount lustre-MDT0000 complete [ 1732.699807] LustreError: 105932:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825350 with bad export cookie 4272075591088690529 [ 1732.704076] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1732.765445] LustreError: 108624:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1732.767986] LustreError: 108624:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1732.853938] Lustre: server umount lustre-MDT0001 complete [ 1736.776854] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1737.915929] Key type lgssc unregistered [ 1738.040314] LNet: 109155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1738.044074] LNetError: 109155:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1738.052334] LNet: Removed LNI 192.168.201.124@tcp [ 1738.375366] Key type .llcrypt unregistered [ 1738.376591] Key type ._llcrypt unregistered [ 1745.440976] Key type ._llcrypt registered [ 1745.442394] Key type .llcrypt registered [ 1745.485405] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1745.899569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1745.914830] alg: No test for adler32 (adler32-zlib) [ 1746.771704] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1746.865619] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1748.440152] Key type lgssc registered [ 1748.850760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1751.086564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1751.086573] LustreError: 110197:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1751.090613] Lustre: Skipped 1 previous similar message [ 1751.090628] LustreError: 110196:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1751.096694] LustreError: 110197:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1751.099674] LustreError: 110197:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1751.104462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.105483] LustreError: 110197:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1752.304487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1755.156728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1755.276664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1756.654748] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1758.319434] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing lsmod [ 1759.386083] LustreError: 111373:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1759.386596] LustreError: 111374:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1759.390764] LustreError: 111373:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4998 [ 1759.393670] LustreError: 111373:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1759.397773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1759.399542] LustreError: 111373:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1759.549342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1760.736846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1760.739311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1760.743442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1764.833927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1764.834254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1764.843383] Lustre: Skipped 1 previous similar message [ 1766.717930] LustreError: 111562:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1766.743513] Lustre: server umount lustre-MDT0000 complete [ 1768.038534] LustreError: 110210:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825385 with bad export cookie 12863808594642170723 [ 1768.039867] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1768.042970] LustreError: 110210:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1768.047416] Lustre: 110228:0:(mgc_request.c:1902:mgc_process_log()) MGC192.168.201.124@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 1768.928226] LustreError: 110669:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f08c4859 x1844262064896896/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1768.939847] LustreError: 111764:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1768.942261] LustreError: 111764:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1769.017512] Lustre: server umount lustre-MDT0001 complete [ 1780.594267] LustreError: 111964:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1780.598442] LustreError: 111964:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1780.662378] Lustre: server umount lustre-OST0000 complete [ 1783.405857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1783.556176] LustreError: 112442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1783.564301] LustreError: 112442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1783.581311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1784.945163] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1787.942630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1788.086570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1789.483889] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1791.535303] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1792.832473] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1795.194704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1797.196170] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1799.037601] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1809.638704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1818.080691] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1818.082764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1818.087436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1818.088784] Lustre: Skipped 2 previous similar messages [ 1820.131478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1820.131915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1820.137709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1822.062282] LustreError: 114710:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1822.063976] LustreError: 114710:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1822.135148] Lustre: server umount lustre-OST0000 complete [ 1824.224809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1824.228070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1824.231952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1824.234274] Lustre: Skipped 1 previous similar message [ 1829.500858] LustreError: 114911:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1829.503277] LustreError: 114911:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1829.531689] Lustre: server umount lustre-MDT0000 complete [ 1830.369223] LustreError: 112442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1830.373330] LustreError: 112442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1830.770843] LustreError: 112422:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825448 with bad export cookie 12863808594642171402 [ 1830.773233] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1830.774925] LustreError: 112422:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1830.921044] Lustre: server umount lustre-MDT0001 complete [ 1834.771287] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1835.780503] Key type lgssc unregistered [ 1835.919405] LNet: 115643:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.922239] LNetError: 115643:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1835.931849] LNet: Removed LNI 192.168.201.124@tcp [ 1836.255115] Key type .llcrypt unregistered [ 1836.256611] Key type ._llcrypt unregistered [ 1845.441387] Key type ._llcrypt registered [ 1845.442408] Key type .llcrypt registered [ 1845.479702] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1845.928673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1845.938474] alg: No test for adler32 (adler32-zlib) [ 1846.788220] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1846.869064] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1848.440129] Key type lgssc registered [ 1848.799833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1851.401215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1851.409417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1852.567596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.848709] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1856.730692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1856.848699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1858.176635] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1860.048203] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1861.321370] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1863.583148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1863.694768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1865.551692] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1867.363334] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1878.055403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1880.060180] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1880.872068] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1882.080911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1882.084656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.089032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1883.107436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1883.107652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1883.108095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1887.855132] LustreError: 119385:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1887.918279] Lustre: server umount lustre-OST0000 complete [ 1892.832820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1892.835341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.839557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1892.841518] Lustre: Skipped 1 previous similar message [ 1895.291913] LustreError: 119591:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1895.294342] LustreError: 119591:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1895.319255] Lustre: server umount lustre-MDT0000 complete [ 1896.493990] LustreError: 116997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825514 with bad export cookie 9659358840904764273 [ 1896.496093] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1896.498171] LustreError: 116997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1896.563800] LustreError: 119793:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1896.565981] LustreError: 119793:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1896.635389] Lustre: server umount lustre-MDT0001 complete [ 1900.213920] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1901.248331] Key type lgssc unregistered [ 1901.393550] LNet: 120323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.397286] LNetError: 120323:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.406569] LNet: Removed LNI 192.168.201.124@tcp [ 1901.725568] Key type .llcrypt unregistered [ 1901.726537] Key type ._llcrypt unregistered [ 1908.599162] Key type ._llcrypt registered [ 1908.600154] Key type .llcrypt registered [ 1908.635803] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1909.130559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1909.151594] alg: No test for adler32 (adler32-zlib) [ 1910.007841] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1910.102782] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1911.696165] Key type lgssc registered [ 1912.049072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1914.500609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1914.508291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1915.654260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1916.915507] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1919.852500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1919.982488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1921.244658] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1923.074956] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1924.308849] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1926.601321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1926.714296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.512337] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1930.293726] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1941.031128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1945.056856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1945.060276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.064139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1946.082733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1946.083388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1946.083494] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.084609] Lustre: Skipped 1 previous similar message [ 1949.807111] LustreError: 123770:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1949.869464] Lustre: server umount lustre-OST0000 complete [ 1951.202129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.202979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1951.208049] Lustre: Skipped 1 previous similar message [ 1956.320973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1956.322882] Lustre: Skipped 2 previous similar messages [ 1957.312904] LustreError: 123973:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1957.316847] LustreError: 123973:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1957.345924] Lustre: server umount lustre-MDT0000 complete [ 1958.510044] LustreError: 121486:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825576 with bad export cookie 4435665925071764993 [ 1958.512745] LustreError: 121486:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1958.513018] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1958.560570] LustreError: 124176:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1958.563559] LustreError: 124176:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1958.641509] Lustre: server umount lustre-MDT0001 complete [ 1962.254259] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1963.278332] Key type lgssc unregistered [ 1963.412840] LNet: 124707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.417524] LNetError: 124707:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1963.426473] LNet: Removed LNI 192.168.201.124@tcp [ 1963.746359] Key type .llcrypt unregistered [ 1963.747890] Key type ._llcrypt unregistered [ 1972.509542] Key type ._llcrypt registered [ 1972.511057] Key type .llcrypt registered [ 1972.548554] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1972.848086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1972.896834] alg: No test for adler32 (adler32-zlib) [ 1973.735427] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1973.808707] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1975.384146] Key type lgssc registered [ 1975.757354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1978.255742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1978.262962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1979.423926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.762482] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1983.655933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1983.771949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1985.131175] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1987.071722] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1988.402595] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1990.746893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1990.858642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1992.731336] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1994.522445] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2005.224468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 2012.777063] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2014.182307] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2014.185408] Lustre: Skipped 2 previous similar messages [ 2015.696705] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2015.700179] Lustre: Skipped 2 previous similar messages [ 2018.297816] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2018.310253] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2024.794935] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.24@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2024.798257] Lustre: Skipped 2 previous similar messages [ 2034.681512] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.24@tcp 192.168.0.[2,10]@tcp [ 2036.797117] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.24/24@tcp in log lustre-MDT0000 [ 2036.801206] Lustre: Skipped 2 previous similar messages [ 2043.895164] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.24/24@tcp [ 2043.897457] Lustre: Skipped 1 previous similar message [ 2046.434807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2046.435726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2046.440245] Lustre: Skipped 1 previous similar message [ 2046.446196] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2051.555079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2051.557904] Lustre: Skipped 2 previous similar messages [ 2052.334669] LustreError: 130138:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2052.405530] Lustre: server umount lustre-OST0000 complete [ 2055.649046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2055.651217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.655045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2056.674082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2059.903489] LustreError: 130340:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2059.906087] LustreError: 130340:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2059.932800] Lustre: server umount lustre-MDT0000 complete [ 2061.287249] LustreError: 127698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825678 with bad export cookie 3424491960863589440 [ 2061.290086] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2061.292515] LustreError: 127698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2061.364032] LustreError: 130543:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2061.366622] LustreError: 130543:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2061.454804] Lustre: server umount lustre-MDT0001 complete [ 2065.550154] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2066.722686] Key type lgssc unregistered [ 2066.866431] LNet: 131074:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2066.870497] LNetError: 131074:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2066.880314] LNet: Removed LNI 192.168.201.124@tcp [ 2067.210179] Key type .llcrypt unregistered [ 2067.211352] Key type ._llcrypt unregistered [ 2078.043903] Key type ._llcrypt registered [ 2078.045575] Key type .llcrypt registered [ 2078.083345] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2078.421246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2078.449370] alg: No test for adler32 (adler32-zlib) [ 2079.306341] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 2079.385305] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2080.960124] Key type lgssc registered [ 2081.297726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2083.776612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2083.782482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.930640] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2084.934553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2084.940792] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.24/24@tcp [ 2086.206100] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2089.022331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2089.134426] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2089.144779] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.24/24@tcp [ 2089.160801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2090.543049] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2092.376294] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2093.655585] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2095.897614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2096.010906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.874595] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2099.656144] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2110.376113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2119.648843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2119.651750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.657059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2120.674105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2120.674437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.674824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2124.398176] LustreError: 134916:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2124.465143] Lustre: server umount lustre-OST0000 complete [ 2130.402615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2130.405639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.412526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2130.414811] Lustre: Skipped 1 previous similar message [ 2136.033134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2136.035898] Lustre: Skipped 2 previous similar messages [ 2140.128123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2140.163497] LustreError: 135118:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2140.165601] LustreError: 135118:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2140.191200] Lustre: server umount lustre-MDT0000 complete [ 2141.152842] LustreError: 132591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2141.159266] LustreError: 132591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2141.431430] LustreError: 132572:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825759 with bad export cookie 17627770387677842277 [ 2141.434892] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2141.436127] LustreError: 132572:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2141.506049] LustreError: 135320:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2141.508721] LustreError: 135320:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2141.589882] Lustre: server umount lustre-MDT0001 complete [ 2145.344600] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2146.447585] Key type lgssc unregistered [ 2146.580534] LNet: 135851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2146.583749] LNetError: 135851:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2146.591477] LNet: Removed LNI 192.168.201.124@tcp [ 2146.930905] Key type .llcrypt unregistered [ 2146.931942] Key type ._llcrypt unregistered [ 2155.676962] Key type ._llcrypt registered [ 2155.678472] Key type .llcrypt registered [ 2155.713888] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 2161.956572] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2162.405023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2162.417472] alg: No test for adler32 (adler32-zlib) [ 2163.269316] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 2163.353760] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2164.936234] Key type lgssc registered [ 2165.301542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2168.269804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2170.524396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2172.504142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2174.614668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2176.862737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2176.882363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2176.887960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.966218] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2177.978285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2178.014496] Lustre: lustre-MDT0000: new disk, initializing [ 2178.040037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2178.046924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2179.309270] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2183.441524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2183.464202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2183.494272] Lustre: 139197:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2183.516463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2183.518188] Lustre: Skipped 1 previous similar message [ 2183.549875] Lustre: lustre-MDT0001: new disk, initializing [ 2183.580871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2183.591877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2183.595309] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2184.824598] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2187.964575] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2189.272715] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2191.547643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2191.577040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2191.665828] Lustre: lustre-OST0000: new disk, initializing [ 2191.667503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2191.694506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2193.607378] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2196.661845] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2198.045721] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2198.519828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2198.523402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2198.554815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2199.139257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2200.388752] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2200.456802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2203.618493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2203.619386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2203.620204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2203.620971] LustreError: Skipped 1 previous similar message [ 2203.624321] Lustre: Skipped 1 previous similar message [ 2207.146381] LustreError: 142141:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2207.199969] Lustre: server umount lustre-OST0000 complete [ 2208.737278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2208.737817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2208.740473] Lustre: Skipped 1 previous similar message [ 2208.741857] Lustre: Skipped 1 previous similar message [ 2213.857165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2213.860657] Lustre: Skipped 2 previous similar messages [ 2214.523166] LustreError: 142342:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2214.525699] LustreError: 142342:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2214.554635] Lustre: server umount lustre-MDT0000 complete [ 2215.757426] LustreError: 139187:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825833 with bad export cookie 10563960981009177440 [ 2215.760488] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2215.761748] LustreError: 139187:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2215.820688] LustreError: 142544:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2215.823433] LustreError: 142544:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2215.901334] Lustre: server umount lustre-MDT0001 complete [ 2218.742300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.912255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2220.205745] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2223.143655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2223.255173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2224.588724] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2226.400488] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2227.703849] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2230.001721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2231.948661] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2233.741200] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2235.173294] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2244.457726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2245.822721] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2245.907898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2249.298475] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2249.325165] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2249.387487] Lustre: lustre-OST0001: new disk, initializing [ 2249.390760] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2249.420128] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2249.422420] Lustre: Skipped 1 previous similar message [ 2251.282411] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2254.537466] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2257.403958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2257.406517] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2257.436793] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2257.661074] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2258.960803] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2259.039567] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2260.337268] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cec84675000.ost_server_uuid 50 [ 2260.906682] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8cec84675000.ost_server_uuid in FULL state after 0 sec [ 2267.621291] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2267.621469] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2267.623742] LustreError: Skipped 1 previous similar message [ 2267.624293] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2267.628049] Lustre: Skipped 1 previous similar message [ 2267.633539] Lustre: Skipped 1 previous similar message [ 2272.741589] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2272.743702] Lustre: Skipped 1 previous similar message [ 2277.344148] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2277.366871] LustreError: 147356:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2277.368523] LustreError: 147356:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2277.386806] Lustre: server umount lustre-OST0001 complete [ 2277.860414] LustreError: 144619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2277.866798] LustreError: 144619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2279.393171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2279.393277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.395802] LustreError: Skipped 1 previous similar message [ 2279.401721] Lustre: Skipped 1 previous similar message [ 2282.979721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2282.979775] LustreError: 144618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2282.982229] Lustre: Skipped 3 previous similar messages [ 2282.986763] LustreError: 144618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2288.101866] LustreError: 144618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2288.110275] LustreError: 144618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2293.216146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2293.221445] LustreError: 147269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2293.227960] LustreError: 147269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2293.246074] LustreError: 147559:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2293.248411] LustreError: 147559:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2293.306210] Lustre: server umount lustre-OST0000 complete [ 2295.264822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2295.267278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.760297] Lustre: server umount lustre-MDT0000 complete [ 2302.024225] LustreError: 146089:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825919 with bad export cookie 10563960981009178308 [ 2302.026380] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2302.030785] LustreError: 146089:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2302.097130] LustreError: 147963:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2302.099662] LustreError: 147963:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2302.182929] Lustre: server umount lustre-MDT0001 complete [ 2308.835668] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2309.902844] Key type lgssc unregistered [ 2310.042990] LNet: 148950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2310.046197] LNetError: 148950:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2310.054426] LNet: Removed LNI 192.168.201.124@tcp [ 2310.363855] Key type .llcrypt unregistered [ 2310.365461] Key type ._llcrypt unregistered [ 2321.200603] Key type ._llcrypt registered [ 2321.201792] Key type .llcrypt registered [ 2321.237953] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 2327.709777] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2328.093746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2328.113070] alg: No test for adler32 (adler32-zlib) [ 2328.973044] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 2329.061698] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2330.640136] Key type lgssc registered [ 2331.006787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2334.136939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2336.490511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2338.625921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2340.906906] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2343.418698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2343.444102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2343.448926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.539759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2344.554456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2344.596150] Lustre: lustre-MDT0000: new disk, initializing [ 2344.624061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2344.631872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2346.059904] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2350.443978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2350.468094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2350.503626] Lustre: 152872:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2350.532552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2350.535481] Lustre: Skipped 1 previous similar message [ 2350.580462] Lustre: lustre-MDT0001: new disk, initializing [ 2350.619263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2350.632742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2350.637609] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2351.977742] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2355.302915] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2356.668683] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2359.096116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2359.124480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2359.229639] Lustre: lustre-OST0000: new disk, initializing [ 2359.234136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2359.270988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2361.092339] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2361.846442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2361.850625] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2361.878367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2364.100719] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2365.454019] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2365.547795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2366.781511] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2366.853598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2372.066462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.067300] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2372.068169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2372.070192] Lustre: Skipped 1 previous similar message [ 2372.075952] LustreError: Skipped 1 previous similar message [ 2373.549344] LustreError: 155808:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2373.604082] Lustre: server umount lustre-OST0000 complete [ 2376.160836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2376.162561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.166088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2376.167682] Lustre: Skipped 1 previous similar message [ 2377.185128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2377.185714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2380.924265] LustreError: 156010:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2380.926180] LustreError: 156010:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2380.948879] Lustre: server umount lustre-MDT0000 complete [ 2382.213438] LustreError: 153652:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825999 with bad export cookie 2718858461109753005 [ 2382.217377] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2382.217794] LustreError: 153652:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2382.285433] LustreError: 156212:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2382.287751] LustreError: 156212:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2382.388364] Lustre: server umount lustre-MDT0001 complete [ 2385.265241] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 14:46:42 (1758826002) [ 2390.714583] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 2394.020497] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2397.693176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2399.902362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2402.038504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2404.323611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2406.788908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2406.810890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2406.911174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2406.923812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2406.964847] Lustre: lustre-MDT0000: new disk, initializing [ 2406.995257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2407.004628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2408.377814] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2412.533844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2412.556444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2412.588742] Lustre: 159199:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2412.609733] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2412.611687] Lustre: Skipped 1 previous similar message [ 2412.649373] Lustre: lustre-MDT0001: new disk, initializing [ 2412.680442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2412.692447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2412.697038] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2413.995169] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2417.247663] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2418.501217] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2420.811053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2420.838637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2420.941982] Lustre: lustre-OST0000: new disk, initializing [ 2420.944894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2420.973188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2422.954793] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2423.096627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2423.099690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2423.135461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2426.112093] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2428.131951] Lustre: Failing over lustre-OST0000 [ 2428.162200] LustreError: 161636:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2428.165302] LustreError: 161636:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2428.227322] Lustre: server umount lustre-OST0000 complete [ 2441.189499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2441.292866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.297256] Lustre: Skipped 1 previous similar message [ 2441.299985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2441.306153] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2442.877245] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2443.318971] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2444.385102] Lustre: 151091:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1758826046/real 1758826046] req@00000000a9988405 x1844262675208192/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1758826062 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2444.397907] Lustre: 151091:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2444.608842] Lustre: Failing over lustre-MDT0000 [ 2444.679768] LustreError: 162608:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2444.681652] LustreError: 162608:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2444.708531] Lustre: server umount lustre-MDT0000 complete [ 2446.305807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2446.308913] Lustre: Skipped 2 previous similar messages [ 2449.376154] Lustre: 151089:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1758826051/real 1758826051] req@00000000a6967d70 x1844262675209984/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1758826067 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2451.426455] LustreError: 159209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.434355] LustreError: 159209:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2453.109180] LustreError: 159204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2454.496267] Lustre: 151090:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1758826056/real 1758826056] req@000000006c524482 x1844262675211136/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1758826072 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2456.546146] LustreError: 159209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.550450] LustreError: 159209:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2457.570094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2457.607310] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2457.612670] Lustre: 159201:0:(mgc_request.c:1902:mgc_process_log()) MGC192.168.201.124@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2457.682843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2457.735924] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2458.229168] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2458.976062] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2463.206593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2463.214439] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2463.218333] Lustre: Skipped 1 previous similar message [ 2463.238829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2468.323929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2468.324219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2468.325317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2468.325322] Lustre: Skipped 1 previous similar message [ 2468.326744] LustreError: Skipped 1 previous similar message [ 2468.330550] Lustre: Skipped 1 previous similar message [ 2470.447108] LustreError: 163496:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2470.449888] LustreError: 163496:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2470.519045] Lustre: server umount lustre-OST0000 complete [ 2473.442638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2473.443280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2473.448890] Lustre: Skipped 2 previous similar messages [ 2483.681404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2483.683106] Lustre: Skipped 3 previous similar messages [ 2486.240143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2486.307378] Lustre: server umount lustre-MDT0000 complete [ 2487.593239] LustreError: 162086:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826105 with bad export cookie 2718858461109755063 [ 2487.596578] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2487.597231] LustreError: 162086:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2487.660874] LustreError: 163899:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2487.662456] LustreError: 163899:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2487.743207] Lustre: server umount lustre-MDT0001 complete [ 2491.518713] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2492.690646] Key type lgssc unregistered [ 2492.828355] LNet: 164430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2492.830579] LNetError: 164430:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2492.837322] LNet: Removed LNI 192.168.201.124@tcp [ 2493.151051] Key type .llcrypt unregistered [ 2493.152019] Key type ._llcrypt unregistered [ 2513.333202] Key type ._llcrypt registered [ 2513.334219] Key type .llcrypt registered [ 2513.373183] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2513.813562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2513.857945] alg: No test for adler32 (adler32-zlib) [ 2514.720241] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 2514.809242] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2516.392195] Key type lgssc registered [ 2516.757913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2519.440322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2519.447569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2520.593838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2521.939602] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2525.071349] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2527.078111] LustreError: 165955:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2527.081597] LustreError: 165955:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2527.084852] LustreError: 165955:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2527.147741] LustreError: 166478:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2527.230958] Lustre: server umount lustre-MDT0000 complete [ 2532.129306] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 2535.275346] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2538.732563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2540.953511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2542.996758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2545.164223] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2547.453370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2547.477968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2547.566906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2547.578940] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2547.626968] Lustre: lustre-MDT0000: new disk, initializing [ 2547.653888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.660995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2548.937183] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2553.036560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2553.058304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2553.088506] Lustre: 169120:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2553.108978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2553.110505] Lustre: Skipped 1 previous similar message [ 2553.153165] Lustre: lustre-MDT0001: new disk, initializing [ 2553.185924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2553.197392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2553.200161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2554.488744] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2557.657245] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2559.000458] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2561.361429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2561.382608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2561.470759] Lustre: lustre-OST0000: new disk, initializing [ 2561.472497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2561.499074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2563.427502] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2564.801854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2564.805332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2564.843149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2566.654723] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2568.525198] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2569.127865] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:49:46 (1758826186) [ 2635.232550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2635.235938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2635.240947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2636.768423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2636.768638] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2636.769983] Lustre: Skipped 1 previous similar message [ 2636.775822] Lustre: Skipped 1 previous similar message [ 2640.760659] LustreError: 172622:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2640.762389] LustreError: 172622:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2640.804207] Lustre: server umount lustre-MDT0000 complete [ 2641.888400] LustreError: 169130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.893328] LustreError: 169130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2642.022389] LustreError: 172521:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826259 with bad export cookie 5389253925136709536 [ 2642.023509] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2642.026587] LustreError: 172521:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2647.008646] LustreError: 169127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2647.008768] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2647.022780] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2647.025487] Lustre: Skipped 1 previous similar message [ 2648.123474] LustreError: 172824:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2648.125254] LustreError: 172824:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2648.190982] Lustre: server umount lustre-MDT0001 complete [ 2655.530472] LustreError: 173025:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2655.532781] LustreError: 173025:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2655.595716] Lustre: server umount lustre-OST0000 complete [ 2658.516444] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 2661.931595] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2665.566853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2667.861478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2669.871204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2672.127409] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2674.308793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2674.331074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2674.417981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2674.432550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2674.473140] Lustre: lustre-MDT0000: new disk, initializing [ 2674.497974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2674.506391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2675.826126] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2680.049751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2680.071954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2680.103928] Lustre: 175359:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2680.126735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2680.128544] Lustre: Skipped 1 previous similar message [ 2680.163169] Lustre: lustre-MDT0001: new disk, initializing [ 2680.208083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2680.211791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2681.612414] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2684.805046] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2686.138243] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2688.557216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2688.584762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2688.682436] Lustre: lustre-OST0000: new disk, initializing [ 2688.684590] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2688.715221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2688.718327] Lustre: Skipped 1 previous similar message [ 2690.621216] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2691.895529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2691.900113] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2691.926541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2693.723399] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2695.106160] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2695.186173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2696.610812] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2696.697419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2699.374130] LustreError: 178299:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2699.377601] LustreError: 178299:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2699.438218] Lustre: server umount lustre-OST0000 complete [ 2700.768909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2700.772585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2700.776736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2707.424967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2707.427707] Lustre: Skipped 3 previous similar messages [ 2710.500270] LustreError: 178499:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2710.502753] LustreError: 178499:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2710.526298] Lustre: server umount lustre-MDT0000 complete [ 2711.736243] LustreError: 175350:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826329 with bad export cookie 5389253925136868352 [ 2711.739201] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2711.740589] LustreError: 175350:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2722.357511] Lustre: server umount lustre-MDT0001 complete [ 2725.309965] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:52:22 (1758826342) [ 2730.605036] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 2733.437385] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2736.636989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2738.880866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2740.931524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2742.971181] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2745.287263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2745.308683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2745.382077] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2745.394326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2745.436170] Lustre: lustre-MDT0000: new disk, initializing [ 2745.459805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2745.469586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2746.838194] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2751.018839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2751.041459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2751.074279] Lustre: 181687:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2751.185261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2752.559413] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2755.728602] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2757.024531] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2759.339626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2759.367040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2760.248731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2760.287933] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2761.409024] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2764.614085] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2770.402870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.403780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2770.404630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2770.406101] Lustre: Skipped 2 previous similar messages [ 2770.412096] LustreError: Skipped 1 previous similar message [ 2772.653094] LustreError: 184232:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2772.655076] LustreError: 184232:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2772.714617] Lustre: server umount lustre-OST0000 complete [ 2788.320148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2788.377985] Lustre: server umount lustre-MDT0000 complete [ 2789.718986] LustreError: 182471:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826407 with bad export cookie 5389253925136869220 [ 2789.725611] LustreError: 182471:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2789.726016] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2789.901585] Lustre: server umount lustre-MDT0001 complete [ 2792.844617] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:53:30 (1758826410) [ 2798.162406] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 2801.558921] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2804.951279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2807.211737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2809.284073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2811.583411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2814.417495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2814.437902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2814.531527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2814.546059] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2814.549150] Lustre: Skipped 3 previous similar messages [ 2814.588162] Lustre: lustre-MDT0000: new disk, initializing [ 2814.590764] Lustre: Skipped 2 previous similar messages [ 2814.617366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2814.620159] Lustre: Skipped 2 previous similar messages [ 2814.626663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2814.631133] Lustre: Skipped 2 previous similar messages [ 2815.965702] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2820.177266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2820.204562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2820.243332] Lustre: 187619:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2820.341490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2821.665617] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2824.868894] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2826.163168] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2828.540452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2828.564891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2829.046278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2830.622478] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2833.696661] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2839.527254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2839.527637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2839.535127] Lustre: Skipped 2 previous similar messages [ 2839.537917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2839.539541] Lustre: Skipped 8 previous similar messages [ 2841.645095] LustreError: 190167:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2841.646867] LustreError: 190167:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2841.704687] Lustre: server umount lustre-OST0000 complete [ 2849.041971] Lustre: server umount lustre-MDT0000 complete [ 2849.761468] LustreError: 187631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2849.768017] LustreError: 187631:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2850.287534] LustreError: 189374:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826467 with bad export cookie 5389253925136870291 [ 2850.291700] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2850.291971] LustreError: 189374:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2850.437080] Lustre: server umount lustre-MDT0001 complete [ 2854.425814] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2855.552350] Key type lgssc unregistered [ 2855.677390] LNet: 191100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2855.679761] LNetError: 191100:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2855.686306] LNet: Removed LNI 192.168.201.124@tcp [ 2855.993238] Key type .llcrypt unregistered [ 2855.994496] Key type ._llcrypt unregistered [ 2865.233457] Key type ._llcrypt registered [ 2865.235255] Key type .llcrypt registered [ 2865.272149] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2865.573407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2865.579420] alg: No test for adler32 (adler32-zlib) [ 2866.432437] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 2866.510774] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2868.088153] Key type lgssc registered [ 2868.451433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2870.981049] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2870.987200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2872.143275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2873.524903] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2876.380568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2876.504191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2877.777533] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2879.547835] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2880.774487] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2883.351456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2883.481080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.499163] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2887.512222] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2897.832357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2912.225284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2912.227310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2912.230686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2913.250791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2913.251773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.252247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2915.629104] LustreError: 194740:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2915.688250] Lustre: server umount lustre-OST0000 complete [ 2917.344761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2917.348046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2917.352148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2917.354113] Lustre: Skipped 1 previous similar message [ 2923.075491] LustreError: 194942:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2923.077581] LustreError: 194942:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2923.110641] Lustre: server umount lustre-MDT0000 complete [ 2923.490064] LustreError: 192475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2923.495863] LustreError: 192475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2924.385386] LustreError: 194092:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826542 with bad export cookie 5691114220548925458 [ 2924.389403] LustreError: 194092:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2924.389747] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2924.453493] LustreError: 195144:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2924.455089] LustreError: 195144:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2924.531681] Lustre: server umount lustre-MDT0001 complete [ 2928.399155] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2929.511242] Key type lgssc unregistered [ 2929.652393] LNet: 195676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2929.655189] LNetError: 195676:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2929.665332] LNet: Removed LNI 192.168.201.124@tcp [ 2929.972886] Key type .llcrypt unregistered [ 2929.973837] Key type ._llcrypt unregistered [ 2939.285362] Key type ._llcrypt registered [ 2939.286298] Key type .llcrypt registered [ 2939.330602] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2939.832431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2939.871991] alg: No test for adler32 (adler32-zlib) [ 2940.736829] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 2940.835194] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2942.416178] Key type lgssc registered [ 2942.817488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2945.681118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2945.689071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2946.857294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2948.161557] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2950.979550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2951.108598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2952.559610] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2954.424343] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2955.609734] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2957.903487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2958.012038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2959.894181] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2961.671586] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2972.327549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2981.857690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2981.859729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2981.863094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2982.532376] Lustre: lustre-OST0000: Not available for connect from 192.168.201.24@tcp (stopping) [ 2982.883974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2982.884948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2985.774148] LustreError: 199318:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2985.834123] Lustre: server umount lustre-OST0000 complete [ 2987.681658] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8cec86dbc800.ost_server_uuid 50 [ 2988.195857] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2998.199193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2998.199914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2998.206106] Lustre: Skipped 1 previous similar message [ 2998.213540] Lustre: Skipped 3 previous similar messages [ 3003.362113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3003.366072] Lustre: Skipped 1 previous similar message [ 3004.367415] LustreError: 199667:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3004.370847] LustreError: 199667:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3004.403468] Lustre: server umount lustre-MDT0000 complete [ 3005.949326] LustreError: 198669:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826623 with bad export cookie 5714562739616339614 [ 3005.953406] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3005.955968] LustreError: 198669:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3006.040709] LustreError: 199869:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3006.043021] LustreError: 199869:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3006.124039] Lustre: server umount lustre-MDT0001 complete [ 3009.294887] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:57:06 (1758826626) [ 3011.953671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3012.114115] LustreError: 200538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3012.123576] LustreError: 200538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3012.146067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3013.690301] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3017.186471] LustreError: 200538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3017.274828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3017.472132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3019.000555] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3021.553891] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3023.163620] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3025.688719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3025.826450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3027.734531] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3029.523440] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3031.775069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3031.800676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3031.855860] Lustre: lustre-OST0001: new disk, initializing [ 3031.857662] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3033.710370] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3036.838518] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3037.032407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3037.182429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3037.186059] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3037.226607] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3043.297403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3043.299518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3043.303061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3047.399547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3047.403500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3049.262109] LustreError: 203745:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3049.263842] LustreError: 203745:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3049.280908] Lustre: server umount lustre-OST0000 complete [ 3051.265525] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3051.345195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3052.515153] LustreError: 202139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.519070] LustreError: 202139:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3052.619880] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3052.708743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3057.641981] LustreError: 202506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3057.646151] LustreError: 202506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3062.757937] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3062.759119] LustreError: 202506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3062.759592] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3062.760862] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3062.760868] Lustre: Skipped 3 previous similar messages [ 3062.761668] LustreError: Skipped 1 previous similar message [ 3062.765243] LustreError: 202506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3068.783088] LustreError: 204446:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3068.785652] LustreError: 204446:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3068.850393] Lustre: server umount lustre-OST0001 complete [ 3076.248499] Lustre: server umount lustre-MDT0000 complete [ 3077.451470] LustreError: 204360:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826695 with bad export cookie 5714562739616340671 [ 3077.455411] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3077.456604] LustreError: 204360:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3077.521904] LustreError: 204852:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3077.523596] LustreError: 204852:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3077.599660] Lustre: server umount lustre-MDT0001 complete [ 3079.572321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3081.185194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3082.451931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3083.995882] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3086.517844] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:58:24 (1758826704) [ 3088.814708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3088.913908] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3088.926260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3089.004385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3089.007508] Lustre: Skipped 1 previous similar message [ 3090.317354] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3093.219191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3093.251587] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3093.264103] Lustre: 206342:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3094.653684] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3096.481497] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3097.693631] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3099.854495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3099.937516] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3101.275631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3101.878556] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3103.667314] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3105.911091] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3105.935596] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3107.800245] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3109.561250] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3111.393429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3111.397367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3111.402879] Lustre: Skipped 3 previous similar messages [ 3111.405192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3111.408821] Lustre: Skipped 5 previous similar messages [ 3116.654292] LustreError: 209537:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3116.656920] LustreError: 209537:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3116.677605] Lustre: server umount lustre-OST0000 complete [ 3120.245932] LustreError: 209454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3120.250447] LustreError: 209454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3130.336842] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3130.338689] LustreError: Skipped 1 previous similar message [ 3133.435849] Lustre: server umount lustre-OST0001 complete [ 3140.850508] Lustre: server umount lustre-MDT0000 complete [ 3141.089723] LustreError: 206353:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3141.099111] LustreError: 206353:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3142.511054] LustreError: 207990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826760 with bad export cookie 5714562739616342057 [ 3142.517509] LustreError: 207990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3142.517604] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3142.693581] Lustre: server umount lustre-MDT0001 complete [ 3145.023809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3146.562861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3147.875462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3149.293649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3151.675350] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:59:29 (1758826769) [ 3157.399180] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 3160.846617] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 3165.144190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3167.387704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3169.262372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3171.326610] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3173.600835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3173.624857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3173.706046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3173.718954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3173.758402] Lustre: lustre-MDT0000: new disk, initializing [ 3173.782629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3173.785930] Lustre: Skipped 3 previous similar messages [ 3173.792980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3175.102637] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3179.289942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3179.311674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3179.342493] Lustre: 213959:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3179.371971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3179.374349] Lustre: Skipped 1 previous similar message [ 3179.415449] Lustre: lustre-MDT0001: new disk, initializing [ 3179.461947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3179.464779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3180.712671] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3183.784408] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3185.055883] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3187.455379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3187.482869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3187.583729] Lustre: lustre-OST0000: new disk, initializing [ 3187.585667] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3187.741102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3187.743767] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3187.775585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3189.615653] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3192.718855] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3194.183336] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3194.297489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3195.864474] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3195.942479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3197.921619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3197.923094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3197.923505] LustreError: Skipped 2 previous similar messages [ 3197.924206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3197.924211] Lustre: Skipped 8 previous similar messages [ 3197.930415] Lustre: Skipped 6 previous similar messages [ 3202.603189] LustreError: 216897:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3202.604795] LustreError: 216897:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3202.665065] Lustre: server umount lustre-OST0000 complete [ 3218.400175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3218.455026] Lustre: server umount lustre-MDT0000 complete [ 3219.716397] LustreError: 213950:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826837 with bad export cookie 5714562739616343604 [ 3219.719381] LustreError: 213950:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3219.719949] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3219.860770] Lustre: server umount lustre-MDT0001 complete [ 3222.876362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3223.050159] LustreError: 217777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3224.471285] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3227.910302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3229.559984] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3231.747983] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3233.220253] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3236.182093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3238.348096] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3239.979710] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3241.163077] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3248.454881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 3249.883391] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3249.989305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3252.971353] Lustre: server umount lustre-OST0000 complete [ 3255.149662] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3255.218354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3256.441838] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3256.534377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3279.995923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3280.105655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.107476] Lustre: Skipped 5 previous similar messages [ 3281.866933] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3283.602385] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3284.153024] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3287.210277] LustreError: 222247:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3287.212527] LustreError: 222247:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3287.279965] Lustre: server umount lustre-OST0000 complete [ 3289.568892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3289.570971] LustreError: Skipped 3 previous similar messages [ 3289.572093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3289.576183] Lustre: Skipped 4 previous similar messages [ 3289.578453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3289.579991] Lustre: Skipped 10 previous similar messages [ 3290.585287] Lustre: server umount lustre-MDT0000 complete [ 3290.594313] LustreError: 218490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3290.602469] LustreError: 218490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3291.817148] LustreError: 217758:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826909 with bad export cookie 5714562739616344472 [ 3291.820086] LustreError: 217758:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3291.820834] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3291.960343] Lustre: server umount lustre-MDT0001 complete [ 3294.788078] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:01:52 (1758826912) [ 3296.940930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3298.369546] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3301.202401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3302.681642] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3304.324301] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3305.560728] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3308.097462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3310.090175] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3311.934280] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3313.280205] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3321.589407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3322.966982] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3323.053419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3325.480271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3325.507158] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3325.566360] Lustre: lustre-OST0001: new disk, initializing [ 3325.568117] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3327.472559] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3331.006217] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3332.340210] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3333.630791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3333.633824] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3333.663250] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3334.464541] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3335.759953] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3335.825974] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3342.525911] Lustre: server umount lustre-OST0001 complete [ 3344.512614] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3344.590615] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3346.003491] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3346.088541] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3369.843367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3371.885611] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3373.632361] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3374.335763] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3381.176225] Lustre: server umount lustre-OST0001 complete [ 3399.275724] Lustre: server umount lustre-OST0000 complete [ 3406.740910] Lustre: server umount lustre-MDT0000 complete [ 3408.164494] LustreError: 223304:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758827025 with bad export cookie 5714562739616345403 [ 3408.170704] LustreError: 223304:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3408.170769] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3414.370598] Lustre: server umount lustre-MDT0001 complete [ 3416.319413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3417.597597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3418.790610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3420.058863] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3422.188919] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 15:03:59 (1758827039) [ 3424.356384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3424.448350] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3424.458293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3424.530900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3424.534478] Lustre: Skipped 5 previous similar messages [ 3425.816063] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3429.004922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3429.047444] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3429.059882] Lustre: 231363:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3430.408066] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3432.143556] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3433.290398] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3435.645168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3435.715972] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3437.550038] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3439.128373] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3448.253176] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3448.275842] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3450.125556] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3452.007212] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3455.718532] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 3457.254982] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3457.329416] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3458.615528] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3458.685207] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3459.941396] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cec91694000.ost_server_uuid 50 [ 3460.481372] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8cec91694000.ost_server_uuid in FULL state after 0 sec [ 3460.809421] Lustre: Permanently deactivating lustre-OST0001 [ 3460.815448] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3461.789923] Lustre: Permanently reactivating lustre-OST0001 [ 3465.702633] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3465.703195] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3465.704637] LustreError: Skipped 5 previous similar messages [ 3465.707290] Lustre: Skipped 7 previous similar messages [ 3465.707858] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3465.710892] Lustre: Skipped 18 previous similar messages [ 3468.971113] LustreError: 235347:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3468.973091] LustreError: 235347:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 3468.990877] Lustre: server umount lustre-OST0001 complete [ 3470.818720] LustreError: 232984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3470.823324] LustreError: 232984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 3476.327297] Lustre: server umount lustre-OST0000 complete [ 3483.604660] Lustre: server umount lustre-MDT0000 complete [ 3485.132154] LustreError: 233902:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758827102 with bad export cookie 5714562739616346663 [ 3485.138077] LustreError: 233902:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3485.139450] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3485.321353] Lustre: server umount lustre-MDT0001 complete [ 3487.409601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3488.768797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3490.130749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3491.554916] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3493.802655] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 15:05:11 (1758827111) [ 3494.185263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3496.413560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3496.509203] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3496.521176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3496.524740] Lustre: Skipped 5 previous similar messages [ 3497.797048] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3500.515287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3500.546519] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3500.555407] Lustre: 237500:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3501.859413] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3503.508099] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3504.597258] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3506.729663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3506.759730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3506.836093] Lustre: Permanently deactivating lustre-OST0000 [ 3506.840531] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3508.462444] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3508.636517] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3510.329621] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3512.487763] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3513.810883] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3514.344435] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3516.018204] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3516.566464] Lustre: Permanently reactivating lustre-OST0000 [ 3521.021249] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3521.023663] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3521.026838] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3521.050183] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3521.059343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3532.539589] Lustre: server umount lustre-OST0001 complete [ 3548.128154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3548.229529] Lustre: server umount lustre-OST0000 complete [ 3556.180078] Lustre: server umount lustre-MDT0000 complete [ 3557.383197] LustreError: 237491:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758827175 with bad export cookie 5714562739616348266 [ 3557.385750] LustreError: 237491:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3557.386389] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3557.523283] Lustre: server umount lustre-MDT0001 complete [ 3561.628477] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 3562.730786] Key type lgssc unregistered [ 3562.862397] LNet: 241892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3562.864588] LNetError: 241892:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3562.873319] LNet: Removed LNI 192.168.201.124@tcp [ 3563.175111] Key type .llcrypt unregistered [ 3563.176618] Key type ._llcrypt unregistered [ 3571.952653] Key type ._llcrypt registered [ 3571.953772] Key type .llcrypt registered [ 3571.985319] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 3572.344627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3572.353251] alg: No test for adler32 (adler32-zlib) [ 3573.198159] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 3573.271959] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 3574.848189] Key type lgssc registered [ 3575.203952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3576.273690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3578.667195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3578.685215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3579.895565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3581.257123] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3584.404872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3584.434854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3584.467739] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3584.476942] Lustre: 243310:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3584.480186] Lustre: Permanently deactivating lustre-MDT0001 [ 3584.485423] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3584.606743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3585.952667] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3587.436731] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3587.439995] LustreError: 243351:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3587.445615] LustreError: 243351:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3587.449354] LustreError: 243351:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 3587.729873] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3588.920882] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3591.175030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3591.276271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3593.092651] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3594.340333] LustreError: 244940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3594.343197] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3594.346078] LustreError: 244940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3595.022446] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3597.361049] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3597.441686] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3599.547500] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3601.393747] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3602.919219] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3604.973761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3604.974270] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3607.146545] Lustre: Permanently reactivating lustre-MDT0001 [ 3607.153398] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3607.155535] Lustre: Skipped 2 previous similar messages [ 3610.366376] Lustre: lustre-MDT0001: Client 9ff49a70-1941-4d2f-92e6-9bb9bbbe8c51 (at 192.168.201.24@tcp) reconnecting [ 3612.512160] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3613.066910] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3614.704271] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3614.774477] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3616.065284] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3616.137737] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3616.925671] Lustre: Permanently deactivating lustre-MDT0001 [ 3616.932258] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3616.934259] Lustre: Skipped 2 previous similar messages [ 3619.132837] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3623.331327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3623.333662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3623.336462] Lustre: Skipped 3 previous similar messages [ 3623.338852] Lustre: Skipped 2 previous similar messages [ 3623.808130] LustreError: 247382:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3623.841120] Lustre: server umount lustre-MDT0000 complete [ 3625.208204] LustreError: 243299:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758827242 with bad export cookie 8692052510241313166 [ 3625.212475] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3626.047822] LustreError: 247584:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3626.049924] LustreError: 247584:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3626.130570] Lustre: server umount lustre-MDT0001 complete [ 3627.488080] LustreError: 247784:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3627.489846] LustreError: 247784:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3627.501983] Lustre: server umount lustre-OST0000 complete [ 3631.654110] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 15:07:29 (1758827249) [ 3635.128173] LustreError: 248788:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3635.129902] LustreError: 248788:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3635.188765] Lustre: server umount lustre-OST0001 complete [ 3637.288165] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 3640.647318] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 3644.343989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3646.919307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3649.150767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3651.654215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3654.173516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3654.195238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3654.278290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3654.280968] Lustre: Skipped 2 previous similar messages [ 3654.290263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3654.327284] Lustre: lustre-MDT0000: new disk, initializing [ 3654.351243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3654.357591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3655.819056] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3659.986434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3660.008634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3660.043721] Lustre: 250969:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3660.067489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3660.070742] Lustre: Skipped 1 previous similar message [ 3660.108474] Lustre: lustre-MDT0001: new disk, initializing [ 3660.149255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3660.152792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3661.513615] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3664.504749] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3665.742868] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3668.689201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3668.718417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3668.853365] Lustre: lustre-OST0000: new disk, initializing [ 3668.856280] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3668.887928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3668.890659] Lustre: Skipped 1 previous similar message [ 3671.094137] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3672.443344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3672.449741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3672.479632] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3674.288040] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3674.991878] LustreError: 250975:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3679.495545] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3679.519570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3679.577851] Lustre: lustre-OST0001: new disk, initializing [ 3679.581042] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3681.688512] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3685.080125] LustreError: 250975:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3685.094819] LustreError: 250975:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3685.347913] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3686.101714] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3689.468098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3689.471504] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3689.499851] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3695.184183] LustreError: 250975:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3696.259607] Lustre: lustre-OST0001: Not available for connect from 192.168.201.24@tcp (stopping) [ 3696.263279] Lustre: Skipped 1 previous similar message [ 3699.686445] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3699.687418] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3699.690507] LustreError: Skipped 1 previous similar message [ 3699.691491] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3699.697742] Lustre: Skipped 1 previous similar message [ 3702.064124] LustreError: 254469:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3702.068376] LustreError: 254469:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3702.097820] Lustre: server umount lustre-OST0001 complete [ 3704.810448] LustreError: 254381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3704.820025] LustreError: 254381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3709.923531] LustreError: 253167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3709.928776] LustreError: 253167:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3715.043265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3715.044462] LustreError: 252701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3715.045153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3715.045159] Lustre: Skipped 2 previous similar messages [ 3715.046366] Lustre: Skipped 1 previous similar message [ 3715.050048] LustreError: 252701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3720.163460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3720.164794] LustreError: 252701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3720.165080] Lustre: Skipped 2 previous similar messages [ 3720.173532] LustreError: 252701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3720.301118] LustreError: 254690:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3720.304222] LustreError: 254690:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3720.365338] Lustre: server umount lustre-OST0000 complete [ 3721.696692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3721.698542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3727.853276] Lustre: server umount lustre-MDT0000 complete [ 3729.272524] LustreError: 251750:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758827346 with bad export cookie 8692052510241315987 [ 3729.275582] LustreError: 251750:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3729.277272] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3735.547401] Lustre: server umount lustre-MDT0001 complete [ 3740.314654] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 3741.426353] Key type lgssc unregistered [ 3741.567351] LNet: 255624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3741.571392] LNetError: 255624:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3741.582285] LNet: Removed LNI 192.168.201.124@tcp [ 3741.879673] Key type .llcrypt unregistered [ 3741.880672] Key type ._llcrypt unregistered [ 3754.081194] Key type ._llcrypt registered [ 3754.082278] Key type .llcrypt registered [ 3754.116435] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 3760.888419] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 3761.293298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3761.302046] alg: No test for adler32 (adler32-zlib) [ 3762.155745] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 3762.235393] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 3763.808151] Key type lgssc registered [ 3764.139730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3767.194305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3769.442141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3771.486847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3774.043885] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3776.360906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3776.381957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3776.386928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3777.471486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3777.483572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3777.519641] Lustre: lustre-MDT0000: new disk, initializing [ 3777.541263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3777.547288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3778.851207] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3783.032426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3783.048426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3783.077636] Lustre: 259565:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3783.100559] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3783.103893] Lustre: Skipped 1 previous similar message [ 3783.148508] Lustre: lustre-MDT0001: new disk, initializing [ 3783.183187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3783.193712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3783.197132] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3784.407919] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3787.458511] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3788.666698] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3790.838580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3790.865708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3790.965199] Lustre: lustre-OST0000: new disk, initializing [ 3790.967755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3790.999029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3792.254611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3792.256940] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3792.300489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3792.870693] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3795.997575] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3797.380440] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3797.460211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3798.842040] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3798.905956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3802.594712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3802.595169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.596472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3802.598019] LustreError: Skipped 1 previous similar message [ 3802.600812] Lustre: Skipped 1 previous similar message [ 3805.612212] LustreError: 262501:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3805.669215] Lustre: server umount lustre-OST0000 complete [ 3807.713766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3807.714560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3807.716929] Lustre: Skipped 1 previous similar message [ 3807.718736] Lustre: Skipped 1 previous similar message [ 3812.833211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3812.834750] Lustre: Skipped 1 previous similar message [ 3812.926894] LustreError: 262702:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3812.929339] LustreError: 262702:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3812.961902] Lustre: server umount lustre-MDT0000 complete [ 3814.253403] LustreError: 260348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758827431 with bad export cookie 9252702234493717275 [ 3814.256810] LustreError: 260348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3814.257770] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3814.325571] LustreError: 262903:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3814.327198] LustreError: 262903:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3814.406243] Lustre: server umount lustre-MDT0001 complete [ 3817.398174] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 15:10:34 (1758827434) [ 3820.000752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3820.206433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3821.592233] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3824.265844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3824.379289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3825.635473] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3827.694315] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3828.885275] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3830.952857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3832.727158] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3834.385418] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3885.539048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3885.539285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3885.544776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3885.547969] Lustre: Skipped 1 previous similar message [ 3888.820153] LustreError: 265919:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3888.823891] LustreError: 265919:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3888.937657] Lustre: server umount lustre-OST0000 complete [ 3890.736468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3895.678387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3895.842190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3895.844681] Lustre: Skipped 1 previous similar message [ 3897.707821] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3899.986144] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3906.083553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3906.087208] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3906.087243] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3906.091821] Lustre: Skipped 1 previous similar message [ 3936.737767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3936.739706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3936.742014] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3936.745921] Lustre: Skipped 2 previous similar messages [ 3939.691070] LustreError: 267581:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3939.693747] LustreError: 267581:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3939.760859] Lustre: server umount lustre-OST0000 complete [ 3941.858415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3941.869743] Lustre: Skipped 1 previous similar message [ 3946.977191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3946.978802] Lustre: Skipped 4 previous similar messages [ 3947.391693] Lustre: server umount lustre-MDT0000 complete [ 3949.135646] LustreError: 263555:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758827566 with bad export cookie 9252702234493718143 [ 3949.141460] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3949.142407] LustreError: 263555:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3949.228298] LustreError: 267984:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3949.230245] LustreError: 267984:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3949.306849] Lustre: server umount lustre-MDT0001 complete [ 3953.936829] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 3955.076312] Key type lgssc unregistered [ 3955.209306] LNet: 268515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3955.211410] LNetError: 268515:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3955.222284] LNet: Removed LNI 192.168.201.124@tcp [ 3955.498152] Key type .llcrypt unregistered [ 3955.499152] Key type ._llcrypt unregistered [ 3965.298544] Key type ._llcrypt registered [ 3965.299669] Key type .llcrypt registered [ 3965.341126] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 3965.850246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3965.867940] alg: No test for adler32 (adler32-zlib) [ 3966.726800] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 3966.818655] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 3968.408131] Key type lgssc registered [ 3968.795209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3972.001791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3972.014792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3973.242711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3974.626721] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3978.141464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3978.312457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3980.069310] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3982.116957] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3983.779554] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3986.633778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3986.788367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3988.944277] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3990.783774] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3998.062595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4013.541800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4013.545951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.552119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4013.556497] Lustre: Skipped 1 previous similar message [ 4016.814853] LustreError: 272931:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4016.874052] Lustre: server umount lustre-OST0000 complete [ 4018.658347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.659828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4018.666076] Lustre: Skipped 2 previous similar messages [ 4018.674993] Lustre: Skipped 1 previous similar message [ 4023.777144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4023.779259] Lustre: Skipped 1 previous similar message [ 4024.587743] LustreError: 273133:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4024.593746] LustreError: 273133:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4024.630483] Lustre: server umount lustre-MDT0000 complete [ 4026.312059] LustreError: 269869:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758827643 with bad export cookie 15150569351807431856 [ 4026.316243] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4026.318097] LustreError: 269869:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4026.386974] LustreError: 273336:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4026.388949] LustreError: 273336:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4026.475966] Lustre: server umount lustre-MDT0001 complete [ 4030.564067] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 4031.648964] Key type lgssc unregistered [ 4031.770382] LNet: 273867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4031.774439] LNetError: 273867:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4031.785313] LNet: Removed LNI 192.168.201.124@tcp [ 4032.104645] Key type .llcrypt unregistered [ 4032.106100] Key type ._llcrypt unregistered [ 4040.226419] Key type ._llcrypt registered [ 4040.227720] Key type .llcrypt registered [ 4040.277742] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4040.703989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4040.764822] alg: No test for adler32 (adler32-zlib) [ 4041.613316] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 4041.697992] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 4043.280128] Key type lgssc registered [ 4043.735175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4046.467427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4046.476041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4047.627949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4049.025498] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4052.495806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4052.653900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4054.135363] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4056.346665] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4057.568515] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4060.121257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4060.280284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4062.302404] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4064.392843] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4072.551526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4083.171513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4083.172920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4083.173822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4083.179583] Lustre: Skipped 1 previous similar message [ 4085.358345] LustreError: 277457:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4085.416235] Lustre: server umount lustre-OST0000 complete [ 4088.289572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4088.293816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4088.297132] Lustre: Skipped 2 previous similar messages [ 4092.730516] LustreError: 277659:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4092.732636] LustreError: 277659:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4092.755752] Lustre: server umount lustre-MDT0000 complete [ 4093.408928] LustreError: 275758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4093.413683] LustreError: 275758:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4093.927094] LustreError: 275027:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758827711 with bad export cookie 10881717940626306344 [ 4093.930163] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4093.930868] LustreError: 275027:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4093.999502] LustreError: 277860:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4094.001593] LustreError: 277860:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4094.073365] Lustre: server umount lustre-MDT0001 complete [ 4097.487628] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 4098.487549] Key type lgssc unregistered [ 4098.605279] LNet: 278392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4098.607311] LNetError: 278392:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4098.614281] LNet: Removed LNI 192.168.201.124@tcp [ 4098.886286] Key type .llcrypt unregistered [ 4098.887958] Key type ._llcrypt unregistered [ 4107.806739] Key type ._llcrypt registered [ 4107.807834] Key type .llcrypt registered [ 4107.840584] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4108.121143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4108.129896] alg: No test for adler32 (adler32-zlib) [ 4108.983702] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 4109.065591] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 4110.640121] Key type lgssc registered [ 4110.976663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4113.357343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4113.363486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4114.521965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4115.749761] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4118.705049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4118.841974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4120.308454] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4122.818538] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4124.408378] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4126.928310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4127.040907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4129.091328] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4131.175321] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4139.374356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4149.730709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4149.730920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4149.733992] LustreError: Skipped 1 previous similar message [ 4149.738439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4149.742233] Lustre: Skipped 1 previous similar message [ 4154.853057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4154.856369] Lustre: Skipped 1 previous similar message [ 4154.871742] LustreError: 282805:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4154.940684] Lustre: server umount lustre-OST0000 complete [ 4159.970055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4159.970570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4159.970844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4159.973230] Lustre: Skipped 2 previous similar messages [ 4162.693230] LustreError: 283007:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4162.698070] LustreError: 283007:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4162.735208] Lustre: server umount lustre-MDT0000 complete [ 4164.073142] LustreError: 281387:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758827781 with bad export cookie 760448574231181902 [ 4164.075836] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4164.078318] LustreError: 281387:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4164.145766] LustreError: 283209:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4164.148832] LustreError: 283209:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4164.231987] Lustre: server umount lustre-MDT0001 complete [ 4167.861285] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 4169.146821] Key type lgssc unregistered [ 4169.279390] LNet: 283741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4169.282869] LNetError: 283741:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4169.295329] LNet: Removed LNI 192.168.201.124@tcp [ 4169.592965] Key type .llcrypt unregistered [ 4169.594553] Key type ._llcrypt unregistered [ 4177.047222] Key type ._llcrypt registered [ 4177.048170] Key type .llcrypt registered [ 4177.086407] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4177.473316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4177.482398] alg: No test for adler32 (adler32-zlib) [ 4178.318881] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 4178.391431] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 4179.968193] Key type lgssc registered [ 4180.396373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4183.348171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4183.356111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4184.540477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.789803] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4188.996304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4189.111446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4190.347682] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4192.409320] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4193.839004] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4196.046221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4196.157061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4198.450170] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4200.366306] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4209.511118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4214.754386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4214.754789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4214.756797] LustreError: Skipped 1 previous similar message [ 4214.761420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4214.764356] Lustre: Skipped 1 previous similar message [ 4216.495128] LustreError: 287373:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4216.558774] Lustre: server umount lustre-OST0000 complete [ 4219.874568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4219.874768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4219.880476] Lustre: Skipped 1 previous similar message [ 4219.882919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4219.885859] Lustre: Skipped 1 previous similar message [ 4223.950840] LustreError: 287575:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4223.954250] LustreError: 287575:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4223.987452] Lustre: server umount lustre-MDT0000 complete [ 4224.993189] LustreError: 284943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4224.999120] LustreError: 284943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4225.206823] LustreError: 284901:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758827842 with bad export cookie 13868746695972822585 [ 4225.211290] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4225.212229] LustreError: 284901:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4225.285601] LustreError: 287776:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4225.287237] LustreError: 287776:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4225.368763] Lustre: server umount lustre-MDT0001 complete [ 4229.318421] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 4230.309184] Key type lgssc unregistered [ 4230.475531] LNet: 288307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4230.479193] LNetError: 288307:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4230.486488] LNet: Removed LNI 192.168.201.124@tcp [ 4230.911891] Key type .llcrypt unregistered [ 4230.913469] Key type ._llcrypt unregistered [ 4234.776112] Key type ._llcrypt registered [ 4234.776961] Key type .llcrypt registered [ 4234.809638] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4240.946944] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 4247.549202] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4247.848511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4247.860643] alg: No test for adler32 (adler32-zlib) [ 4248.714803] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 4248.794735] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 4250.376107] Key type lgssc registered [ 4250.733577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4254.111544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4256.613029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4259.263302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4261.688652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4264.367724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4264.385826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4264.390456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4265.468912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4265.480184] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4265.520978] Lustre: lustre-MDT0000: new disk, initializing [ 4265.545116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4265.550797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4267.075467] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4271.614167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4271.636658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4271.669360] Lustre: 291788:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4271.712226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4271.714531] Lustre: Skipped 1 previous similar message [ 4271.758861] Lustre: lustre-MDT0001: new disk, initializing [ 4271.796111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4271.811964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4271.816262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4273.385830] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4276.894972] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4278.283718] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4280.834294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4280.858622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4280.958953] Lustre: lustre-OST0000: new disk, initializing [ 4280.962329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4280.964059] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4281.002503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4283.370895] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4286.443221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4286.447495] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4286.462025] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4286.990416] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4288.502332] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4288.569353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4289.887738] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4289.948632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4291.556229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4291.556527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4291.564302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4291.566929] Lustre: Skipped 1 previous similar message [ 4292.781059] LustreError: 294734:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4292.842561] Lustre: server umount lustre-OST0000 complete [ 4296.674480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4296.675526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4296.677517] Lustre: Skipped 2 previous similar messages [ 4296.680217] Lustre: Skipped 1 previous similar message [ 4300.155500] LustreError: 294935:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 4300.157763] LustreError: 294935:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4300.187373] Lustre: server umount lustre-MDT0000 complete [ 4301.635970] LustreError: 291779:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758827919 with bad export cookie 1931201963212614237 [ 4301.639818] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4301.640841] LustreError: 291779:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4301.719313] LustreError: 295136:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4301.721295] LustreError: 295136:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4301.808942] Lustre: server umount lustre-MDT0001 complete [ 4304.624898] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 15:18:42 (1758827922) [ 4307.322651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4307.486959] LustreError: 295807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4307.497105] LustreError: 295807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4307.507856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4308.855640] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4311.675202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4311.795222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4313.138751] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4315.046288] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4316.674100] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4319.502149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4321.785672] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4323.647378] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4337.120977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4337.125630] LustreError: Skipped 1 previous similar message [ 4337.128561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4337.136652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4338.147549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4343.086121] LustreError: 298079:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4343.088867] LustreError: 298079:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4343.152180] Lustre: server umount lustre-OST0000 complete [ 4347.873511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4347.875764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4347.877295] LustreError: Skipped 1 previous similar message [ 4347.883553] Lustre: Skipped 1 previous similar message [ 4347.884268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4347.890557] Lustre: Skipped 2 previous similar messages [ 4350.586533] LustreError: 298280:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4350.588319] LustreError: 298280:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4350.620136] Lustre: server umount lustre-MDT0000 complete [ 4351.948372] LustreError: 295787:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758827969 with bad export cookie 1931201963212615119 [ 4351.951267] LustreError: 296530:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4351.951979] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4351.954203] LustreError: 295787:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4352.121954] Lustre: server umount lustre-MDT0001 complete [ 4356.122064] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 4357.369899] Key type lgssc unregistered [ 4357.503299] LNet: 299013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4357.505516] LNetError: 299013:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4357.516393] LNet: Removed LNI 192.168.201.124@tcp [ 4357.821591] Key type .llcrypt unregistered [ 4357.822652] Key type ._llcrypt unregistered [ 4366.943968] Key type ._llcrypt registered [ 4366.945388] Key type .llcrypt registered [ 4366.977200] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 4373.233376] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4373.635842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4373.677402] alg: No test for adler32 (adler32-zlib) [ 4374.524938] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 4374.600948] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 4376.176200] Key type lgssc registered [ 4376.545824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4379.670792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4381.980953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4384.055891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4386.050727] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4387.828878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4389.925310] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4392.723403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4392.753743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4392.764783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4393.900509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4393.919581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4393.973981] Lustre: lustre-MDT0000: new disk, initializing [ 4394.008919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4394.021298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4395.421098] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4399.932035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4399.951573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4399.983552] Lustre: 302790:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4400.009056] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4400.013214] Lustre: Skipped 1 previous similar message [ 4400.056954] Lustre: lustre-MDT0001: new disk, initializing [ 4400.087730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4400.099245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4400.103713] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4401.345359] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4404.342547] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4405.571698] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4408.071305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4408.095632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4408.185272] Lustre: lustre-OST2710: new disk, initializing [ 4408.188084] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4408.218766] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4408.954384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4408.958697] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4409.000611] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4410.089864] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4413.589663] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4416.079551] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4416.118097] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4416.197062] Lustre: lustre-OST03e8: new disk, initializing [ 4416.200577] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4416.248895] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4417.994243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4417.998069] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4418.032401] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4418.664728] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4420.618381] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4423.488716] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4423.561799] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4424.804772] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4424.878167] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4426.189176] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4426.298182] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4427.986921] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4428.077515] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4430.816853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4430.818876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4430.822315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4433.381028] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4433.381733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4433.387859] Lustre: Skipped 2 previous similar messages [ 4433.393195] Lustre: Skipped 3 previous similar messages [ 4435.398152] LustreError: 307130:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4435.435263] Lustre: server umount lustre-MDT0000 complete [ 4436.681790] LustreError: 302783:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758828054 with bad export cookie 1498015729153975157 [ 4436.685057] LustreError: 302783:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4436.685845] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4436.759641] LustreError: 307333:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 4436.762119] LustreError: 307333:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4436.866175] Lustre: server umount lustre-MDT0001 complete [ 4447.667085] LustreError: 307533:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4447.670514] LustreError: 307533:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4447.691195] Lustre: server umount lustre-OST2710 complete [ 4459.129154] LustreError: 307735:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4459.132810] LustreError: 307735:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4459.212966] Lustre: server umount lustre-OST03e8 complete [ 4465.544597] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 4469.448545] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4472.783630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4475.235467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4477.292046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4480.001493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4482.770753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4482.803381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4482.895372] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4482.907208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4482.950830] Lustre: lustre-MDT0000: new disk, initializing [ 4482.972192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4482.978151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4484.196165] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4488.698521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4488.735299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4488.785661] Lustre: 310527:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4488.900163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4490.086288] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4493.666720] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4494.921704] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4497.467723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4497.503429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4497.618810] Lustre: lustre-OST0000: new disk, initializing [ 4497.620386] Lustre: Skipped 1 previous similar message [ 4497.622601] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4497.624469] Lustre: Skipped 2 previous similar messages [ 4497.654630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4497.658635] Lustre: Skipped 1 previous similar message [ 4500.016208] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4501.002958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4501.012224] Lustre: Skipped 1 previous similar message [ 4501.017089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4501.060772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4503.041183] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4504.472381] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4504.539627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4505.800896] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4505.900974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4511.201616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4511.202263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4511.204384] LustreError: Skipped 1 previous similar message [ 4511.208152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4511.212278] Lustre: Skipped 1 previous similar message [ 4512.813044] LustreError: 313468:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4512.815902] LustreError: 313468:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4512.875711] Lustre: server umount lustre-OST0000 complete [ 4514.784777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4514.786633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4514.789321] Lustre: Skipped 1 previous similar message [ 4514.790653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4520.487108] Lustre: server umount lustre-MDT0000 complete [ 4521.727972] LustreError: 310519:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758828139 with bad export cookie 1498015729153976760 [ 4521.730277] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4521.731872] LustreError: 310519:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4521.796616] LustreError: 313870:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4521.799434] LustreError: 313870:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4521.885037] Lustre: server umount lustre-MDT0001 complete [ 4524.830582] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 15:22:22 (1758828142) [ 4525.361308] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4526.008255] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 15:22:23 (1758828143) [ 4527.200967] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4530.244761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4531.934126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4533.518957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4535.082361] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4535.520783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4538.513968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4538.660823] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4538.674920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4538.754084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4540.096356] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4543.534015] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4543.568585] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4543.581851] Lustre: 315995:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4545.058592] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4547.190416] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4548.438090] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4551.019422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4551.063431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4551.165251] Lustre: Denying initial registration attempt from nid 192.168.201.124@tcp, specified as failover [ 4551.170768] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4551.174857] LustreError: 317612:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 4551.178214] LustreError: 317612:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4551.181075] LustreError: 317612:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4551.245496] Lustre: server umount lustre-OST0000 complete [ 4551.247045] LustreError: 317612:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4554.208992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4554.212913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4554.218735] Lustre: Skipped 1 previous similar message [ 4554.222218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4554.225188] Lustre: Skipped 2 previous similar messages [ 4559.934068] LustreError: 317980:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4559.935818] LustreError: 317980:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4559.960247] Lustre: server umount lustre-MDT0000 complete [ 4561.475603] LustreError: 316721:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758828179 with bad export cookie 1498015729153977635 [ 4561.479711] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4561.481782] LustreError: 316721:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4561.647234] Lustre: server umount lustre-MDT0001 complete [ 4565.336456] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 4566.387957] Key type lgssc unregistered [ 4566.509356] LNet: 318713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4566.511550] LNetError: 318713:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4566.519289] LNet: Removed LNI 192.168.201.124@tcp [ 4566.849546] Key type .llcrypt unregistered [ 4566.850511] Key type ._llcrypt unregistered [ 4570.358449] Key type ._llcrypt registered [ 4570.359756] Key type .llcrypt registered [ 4570.390892] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4570.726255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4570.752837] alg: No test for adler32 (adler32-zlib) [ 4571.595385] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 4571.679088] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 4573.256172] Key type lgssc registered [ 4573.645381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4575.856317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4577.156578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4578.353880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4579.543696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4579.881645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4586.231071] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4589.386688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4589.395445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4590.491418] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4590.498972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4590.562414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4591.927271] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4594.953073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4594.984841] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4594.999483] Lustre: 321444:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4595.099608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4596.523817] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4598.382763] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4599.621558] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4602.070609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4602.102822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4602.172677] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4602.238081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4604.096275] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4605.814641] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4610.530355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4610.531016] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4610.534254] LustreError: Skipped 1 previous similar message [ 4610.538036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4610.540911] Lustre: Skipped 1 previous similar message [ 4612.720069] LustreError: 323727:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4612.782764] Lustre: server umount lustre-OST0000 complete [ 4615.651412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4615.652377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4615.658229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4615.664224] Lustre: Skipped 2 previous similar messages [ 4620.556942] LustreError: 323929:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4620.564054] LustreError: 323929:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4620.600343] Lustre: server umount lustre-MDT0000 complete [ 4621.997954] LustreError: 322169:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758828239 with bad export cookie 16998956765926964540 [ 4622.004752] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4622.008206] LustreError: 322169:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4622.089697] LustreError: 324130:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4622.093029] LustreError: 324130:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4622.173638] Lustre: server umount lustre-MDT0001 complete [ 4625.666074] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 4626.757884] Key type lgssc unregistered [ 4626.882329] LNet: 324661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4626.884519] LNetError: 324661:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4626.890297] LNet: Removed LNI 192.168.201.124@tcp [ 4627.204850] Key type .llcrypt unregistered [ 4627.206429] Key type ._llcrypt unregistered [ 4637.143228] Key type ._llcrypt registered [ 4637.144541] Key type .llcrypt registered [ 4637.190350] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4637.497441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4637.545084] alg: No test for adler32 (adler32-zlib) [ 4638.400371] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 4638.486171] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 4640.064200] Key type lgssc registered [ 4640.538704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4643.336940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4643.343332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4644.506621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4645.843327] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4648.872945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4648.986426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4650.597360] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4652.517570] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4653.649115] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4655.849562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4655.958607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4657.937996] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4660.109780] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4674.529431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4674.532674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4674.538787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4675.554376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4675.555697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4675.563408] Lustre: Skipped 1 previous similar message [ 4679.433311] LustreError: 328305:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4679.467115] Lustre: server umount lustre-MDT0000 complete [ 4680.676334] LustreError: 326036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4680.692860] LustreError: 326036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4681.196908] LustreError: 326016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758828298 with bad export cookie 16386987515774500950 [ 4681.202236] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4685.793353] LustreError: 326032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4685.793687] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4685.800681] LustreError: 326032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4685.811345] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4685.815311] Lustre: Skipped 3 previous similar messages [ 4686.816911] LustreError: 326032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4686.821110] LustreError: 326032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4687.302496] LustreError: 328508:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4687.305103] LustreError: 328508:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4687.403935] Lustre: server umount lustre-MDT0001 complete [ 4689.605614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4692.568952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4706.272795] LustreError: 325344:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f8fb667a x1844265097388672/t0(0) o250->MGC192.168.201.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4706.395477] LustreError: 327644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4706.423189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4706.444918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4707.704569] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4710.498759] LustreError: 329319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4710.502977] LustreError: 329319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4710.757490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4710.889129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4712.271761] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4714.031119] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4716.007122] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4717.636269] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4718.905955] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4719.580330] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4721.121473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4721.123827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4721.127349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4721.130463] Lustre: Skipped 2 previous similar messages [ 4722.144765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4725.218869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4725.222670] Lustre: Skipped 2 previous similar messages [ 4727.090120] LustreError: 330813:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4727.093764] LustreError: 330813:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4727.171127] Lustre: server umount lustre-OST0000 complete [ 4730.338168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4730.342571] Lustre: Skipped 2 previous similar messages [ 4734.607308] LustreError: 331014:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4734.613667] LustreError: 331014:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4734.650279] Lustre: server umount lustre-MDT0000 complete [ 4735.457419] LustreError: 329322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4735.464634] LustreError: 329322:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4736.313980] LustreError: 326016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758828353 with bad export cookie 16386987515774508104 [ 4736.320904] LustreError: 326016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4736.321672] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4736.507465] Lustre: server umount lustre-MDT0001 complete [ 4740.543827] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 4741.718988] Key type lgssc unregistered [ 4741.851425] LNet: 331747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4741.854580] LNetError: 331747:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4741.864436] LNet: Removed LNI 192.168.201.124@tcp [ 4742.233683] Key type .llcrypt unregistered [ 4742.235566] Key type ._llcrypt unregistered [ 4752.333356] Key type ._llcrypt registered [ 4752.334607] Key type .llcrypt registered [ 4752.376570] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4752.680211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4752.716038] alg: No test for adler32 (adler32-zlib) [ 4753.564817] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 4753.645627] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 4755.224168] Key type lgssc registered [ 4755.754766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4758.649948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4758.658729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4759.827100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4761.415961] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4764.528892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4764.688497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4766.246506] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4768.171162] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4769.464440] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4770.602485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4770.603585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4770.606044] Lustre: Skipped 1 previous similar message [ 4774.689401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4774.691081] Lustre: Skipped 2 previous similar messages [ 4776.712908] LustreError: 334509:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4776.747459] Lustre: server umount lustre-MDT0000 complete [ 4778.073892] LustreError: 334512:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758828395 with bad export cookie 16389722212835749195 [ 4778.077217] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4778.078906] LustreError: 334512:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4778.141681] LustreError: 334714:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4778.144402] LustreError: 334714:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4778.221274] Lustre: server umount lustre-MDT0001 complete [ 4784.167565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4784.305012] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4784.322933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4784.423592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4785.699419] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4788.657962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4788.688241] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4788.700021] Lustre: 335482:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4788.795348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4790.450467] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4792.728816] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4793.915310] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4797.191372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4797.298084] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4797.302475] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4797.307714] LustreError: 337141:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4797.311742] LustreError: 337141:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4797.314292] LustreError: 337141:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4797.369746] Lustre: server umount lustre-OST0000 complete [ 4797.371088] LustreError: 337141:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4800.444993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4800.559950] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4800.671547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4801.715339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 4802.967513] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4804.803778] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4807.566648] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4807.587680] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4807.608252] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4807.636603] Lustre: lustre-OST0001: new disk, initializing [ 4807.638274] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4809.583171] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4811.323502] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4812.562168] LustreError: 339253:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4812.565490] LustreError: 339253:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4812.584670] Lustre: server umount lustre-OST0001 complete [ 4814.328974] LustreError: 337609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4814.332695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4814.333965] LustreError: 337609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4814.335534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4814.344563] Lustre: Skipped 1 previous similar message [ 4819.430874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4819.432393] LustreError: 337610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4819.433943] Lustre: Skipped 2 previous similar messages [ 4819.441965] LustreError: 337610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4820.078124] LustreError: 339453:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4820.080757] LustreError: 339453:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4820.146222] Lustre: server umount lustre-OST0000 complete [ 4824.548280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4824.548579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4824.549541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4824.563778] Lustre: Skipped 1 previous similar message [ 4834.786060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4834.790038] Lustre: Skipped 4 previous similar messages [ 4836.320139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4836.373238] LustreError: 339662:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup [ 4836.376994] LustreError: 339662:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4836.412182] Lustre: server umount lustre-MDT0000 complete [ 4837.915159] LustreError: 335473:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758828455 with bad export cookie 16389722212835749734 [ 4837.917749] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4837.918394] LustreError: 335473:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4838.051231] Lustre: server umount lustre-MDT0001 complete [ 4842.433176] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 4843.455978] Key type lgssc unregistered [ 4843.582434] LNet: 340396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4843.586105] LNetError: 340396:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4843.593397] LNet: Removed LNI 192.168.201.124@tcp [ 4844.065708] Key type .llcrypt unregistered [ 4844.068909] Key type ._llcrypt unregistered [ 4855.249114] Key type ._llcrypt registered [ 4855.250140] Key type .llcrypt registered [ 4855.286991] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 4861.724135] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4862.124775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4862.147103] alg: No test for adler32 (adler32-zlib) [ 4863.002511] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 4863.083286] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 4864.664225] Key type lgssc registered [ 4865.231518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4868.187872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4870.512913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4872.492152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4874.658846] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4877.356533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4877.386903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4877.393295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4878.495130] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4878.510010] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4878.551562] Lustre: lustre-MDT0000: new disk, initializing [ 4878.584526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4878.592736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4880.330046] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4885.505475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4885.534765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4885.579096] Lustre: 344317:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4885.653184] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4885.657278] Lustre: Skipped 1 previous similar message [ 4885.706657] Lustre: lustre-MDT0001: new disk, initializing [ 4885.750421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4885.767790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4885.774146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4887.354872] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4890.754404] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4892.127500] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4895.075450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4895.119169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4895.295696] Lustre: lustre-OST0000: new disk, initializing [ 4895.299311] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4895.354231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4897.669371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4897.672275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4897.713766] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4898.062572] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4901.602748] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4903.214428] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4903.279374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4904.660594] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4904.742599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4908.001955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4908.004110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4908.005697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4908.009021] Lustre: Skipped 1 previous similar message [ 4911.600475] LustreError: 347255:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4911.697950] Lustre: server umount lustre-OST0000 complete [ 4913.121923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4913.122707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4913.125227] Lustre: Skipped 1 previous similar message [ 4913.128127] Lustre: Skipped 2 previous similar messages [ 4918.241682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4918.243265] Lustre: Skipped 1 previous similar message [ 4919.111193] LustreError: 347457:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4919.115231] LustreError: 347457:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4919.165326] Lustre: server umount lustre-MDT0000 complete [ 4920.485396] LustreError: 347459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758828538 with bad export cookie 10283384092726113427 [ 4920.488217] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4920.490886] LustreError: 347459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4920.548654] LustreError: 347660:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4920.550281] LustreError: 347660:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4920.652514] Lustre: server umount lustre-MDT0001 complete [ 4923.869197] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 15:29:01 (1758828541) [ 4926.440022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4929.003137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4938.494307] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 4942.280780] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4946.677499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4949.466557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4952.325341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4954.851354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4957.740495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4957.762787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4957.837397] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4957.849222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4957.886798] Lustre: lustre-MDT0000: new disk, initializing [ 4957.909387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4957.914950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4959.132728] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4963.280735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4963.304451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4963.332720] Lustre: 351430:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4963.352459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4963.354571] Lustre: Skipped 1 previous similar message [ 4963.391956] Lustre: lustre-MDT0001: new disk, initializing [ 4963.425657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4963.439974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4963.442700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4964.885347] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4968.337024] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4969.637456] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4972.112101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4972.143148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4972.283149] Lustre: lustre-OST0000: new disk, initializing [ 4972.286611] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4972.328288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4974.306047] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4977.678580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4977.684242] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4977.735463] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4977.940634] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4979.414267] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4979.480343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4981.179672] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4981.308162] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4982.757043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4982.757125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4982.758293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4982.767557] Lustre: Skipped 1 previous similar message [ 4982.769571] LustreError: Skipped 2 previous similar messages [ 4984.237098] LustreError: 354364:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4984.240895] LustreError: 354364:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4984.310960] Lustre: server umount lustre-OST0000 complete [ 4987.873587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4987.874357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4987.879010] Lustre: Skipped 1 previous similar message [ 4987.880566] Lustre: Skipped 1 previous similar message [ 4998.113109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4998.115499] Lustre: Skipped 4 previous similar messages [ 5000.160171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5000.206568] LustreError: 354565:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5000.208536] LustreError: 354565:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5000.242192] Lustre: server umount lustre-MDT0000 complete [ 5001.818579] LustreError: 351420:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758828619 with bad export cookie 10283384092726114309 [ 5001.821785] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5001.823507] LustreError: 351420:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5001.974840] Lustre: server umount lustre-MDT0001 complete [ 5005.354344] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:30:22 (1758828622) [ 5008.351622] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:30:26 (1758828626) [ 5012.248313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5012.480896] LustreError: 355782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5012.489980] LustreError: 355782:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5012.516864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5013.694264] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5016.855376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5018.376656] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5020.373233] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5021.859574] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5024.449091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5026.780258] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5028.584250] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5042.769674] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5043.333633] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5043.881496] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5044.478481] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5045.195522] Lustre: Failing over lustre-MDT0000 [ 5045.382532] LustreError: 358243:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5045.384836] LustreError: 358243:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5045.410130] Lustre: server umount lustre-MDT0000 complete [ 5047.776992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5047.779433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5047.784705] LustreError: 355781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5049.466917] LustreError: 357949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5049.472058] LustreError: 357949:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5053.409876] LustreError: 355781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5053.414020] LustreError: 355781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5058.529686] LustreError: 355781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5058.533967] LustreError: 355781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5058.598202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5058.636253] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5058.734966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5058.736966] Lustre: Skipped 2 previous similar messages [ 5058.763575] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5059.703173] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5060.100195] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5064.169592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5064.179609] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5064.201971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5065.559273] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5066.206967] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5068.261605] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5068.903235] Lustre: Failing over lustre-MDT0000 [ 5069.089933] LustreError: 359323:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5069.091940] LustreError: 359323:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5069.123775] Lustre: server umount lustre-MDT0000 complete [ 5069.281830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5069.282960] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5069.283915] LustreError: 355781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5069.296381] Lustre: Skipped 4 previous similar messages [ 5082.157789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5082.194979] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5082.310329] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5083.681297] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5085.301551] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5087.729658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5087.731923] Lustre: Skipped 2 previous similar messages [ 5087.740290] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5087.756951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5088.993730] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5089.819947] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5092.834599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5092.835142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5092.835581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5092.838861] Lustre: Skipped 2 previous similar messages [ 5098.388848] Lustre: server umount lustre-MDT0000 complete [ 5099.859910] LustreError: 355807:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758828717 with bad export cookie 10283384092726116976 [ 5099.868292] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5100.061640] Lustre: server umount lustre-MDT0001 complete [ 5112.117083] LustreError: 360755:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5112.120156] LustreError: 360755:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5112.217215] Lustre: server umount lustre-OST0000 complete [ 5118.234522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5118.398051] LustreError: 361581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5118.402523] LustreError: 361581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 5119.670198] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5123.069372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5123.190303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5123.191974] Lustre: Skipped 2 previous similar messages [ 5124.695213] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5126.628295] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5128.199640] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5131.067457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5133.318074] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5135.497400] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5144.485715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5148.641857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5148.644832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5148.650279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5148.653852] Lustre: Skipped 5 previous similar messages [ 5153.787519] Lustre: server umount lustre-OST0000 complete [ 5161.257415] Lustre: server umount lustre-MDT0000 complete [ 5162.634072] LustreError: 361561:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758828780 with bad export cookie 10283384092726117683 [ 5162.637507] LustreError: 361561:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5162.638235] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5162.769918] Lustre: server umount lustre-MDT0001 complete [ 5166.652423] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 5167.962327] Key type lgssc unregistered [ 5168.105398] LNet: 364784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5168.107957] LNetError: 364784:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5168.117470] LNet: Removed LNI 192.168.201.124@tcp [ 5168.453796] Key type .llcrypt unregistered [ 5168.455269] Key type ._llcrypt unregistered [ 5179.278528] Key type ._llcrypt registered [ 5179.279709] Key type .llcrypt registered [ 5179.325716] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5179.837424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5179.855865] alg: No test for adler32 (adler32-zlib) [ 5180.716146] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 5180.801878] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 5182.384176] Key type lgssc registered [ 5182.888794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5186.349222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5186.363731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5187.620138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5189.110158] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5192.493885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5192.652958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5194.143968] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5196.345153] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5197.772625] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5200.293418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5200.455375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5202.690864] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5204.741894] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5212.776106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5223.400774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5223.401834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5223.403865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5223.408291] Lustre: Skipped 2 previous similar messages [ 5225.607944] LustreError: 368525:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5225.649462] Lustre: server umount lustre-MDT0000 complete [ 5227.383831] LustreError: 366239:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758828845 with bad export cookie 2640330463231883112 [ 5227.391469] LustreError: 366239:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5227.396204] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5227.487988] LustreError: 368727:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5227.492035] LustreError: 368727:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5227.616362] Lustre: server umount lustre-MDT0001 complete [ 5239.538323] LustreError: 368928:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5239.540119] LustreError: 368928:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5239.616400] Lustre: server umount lustre-OST0000 complete [ 5244.772216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5244.950738] LustreError: 369658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5244.956426] LustreError: 369658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5244.974464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5246.354518] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5249.571816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5249.694471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5251.375293] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5253.697127] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5254.880852] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5258.166870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5258.314721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5260.734370] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5263.017967] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5269.613197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5285.349681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.352388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5285.357298] Lustre: Skipped 2 previous similar messages [ 5285.360959] Lustre: Skipped 3 previous similar messages [ 5286.990162] LustreError: 372025:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5286.994543] LustreError: 372025:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5287.033150] Lustre: server umount lustre-MDT0000 complete [ 5288.692804] LustreError: 369639:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758828906 with bad export cookie 2640330463231884211 [ 5288.697531] LustreError: 369639:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5288.697566] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5288.851098] Lustre: server umount lustre-MDT0001 complete [ 5300.920379] LustreError: 372428:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5300.924474] LustreError: 372428:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5300.998698] Lustre: server umount lustre-OST0000 complete [ 5307.970112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5308.184037] LustreError: 373256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5308.215204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5309.657264] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5313.505912] LustreError: 373256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5313.656709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5315.505656] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5318.326415] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5320.111732] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5322.960910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5325.731370] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5327.975497] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5333.350666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5338.595381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5338.596397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5338.596876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5338.596882] Lustre: Skipped 1 previous similar message [ 5338.599508] Lustre: Skipped 1 previous similar message [ 5340.588079] LustreError: 375502:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5340.590242] LustreError: 375502:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5340.648723] Lustre: server umount lustre-OST0000 complete [ 5343.713523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5343.714112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5343.716412] Lustre: Skipped 1 previous similar message [ 5343.718366] Lustre: Skipped 1 previous similar message [ 5347.927153] Lustre: server umount lustre-MDT0000 complete [ 5348.833035] LustreError: 373255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5348.836803] LustreError: 373255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5349.082066] LustreError: 373281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758828966 with bad export cookie 2640330463231885219 [ 5349.084937] LustreError: 373281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5349.085605] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5349.212318] Lustre: server umount lustre-MDT0001 complete [ 5352.722150] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 5353.624446] Key type lgssc unregistered [ 5353.743305] LNet: 376436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5353.746221] LNetError: 376436:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5353.755279] LNet: Removed LNI 192.168.201.124@tcp [ 5354.025650] Key type .llcrypt unregistered [ 5354.026617] Key type ._llcrypt unregistered [ 5362.809610] Key type ._llcrypt registered [ 5362.810510] Key type .llcrypt registered [ 5362.841403] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5363.094693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5363.118859] alg: No test for adler32 (adler32-zlib) [ 5363.958449] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 5364.029727] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 5365.600107] Key type lgssc registered [ 5365.922954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5368.226493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5368.234351] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5368.236559] LustreError: 377833:0:(osd_handler.c:8486:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5368.243920] LustreError: 377833:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5368.245640] LustreError: 377833:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5368.247495] LustreError: 377833:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5368.249407] LustreError: 377833:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5370.537285] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5370.539335] LustreError: 378230:0:(osd_handler.c:8486:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5370.546769] LustreError: 378230:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5370.548564] LustreError: 378230:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5370.550385] LustreError: 378230:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5370.552422] LustreError: 378230:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5376.636895] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 5377.579158] Key type lgssc unregistered [ 5377.721346] LNet: 379213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5377.723927] LNetError: 379213:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5377.735302] LNet: Removed LNI 192.168.201.124@tcp [ 5378.016993] Key type .llcrypt unregistered [ 5378.018107] Key type ._llcrypt unregistered [ 5383.532507] Key type ._llcrypt registered [ 5383.533397] Key type .llcrypt registered [ 5383.563898] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 5389.549184] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5389.917969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5389.925377] alg: No test for adler32 (adler32-zlib) [ 5390.760401] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 5390.833081] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 5392.408093] Key type lgssc registered [ 5392.704677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5395.655357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5397.919314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5399.867740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5402.126383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5404.367069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5404.389175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5404.394082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5405.470178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5405.480314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5405.516104] Lustre: lustre-MDT0000: new disk, initializing [ 5405.537965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5405.544186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5406.935421] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5411.125791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5411.152337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5411.183581] Lustre: 382317:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5411.214459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5411.216835] Lustre: Skipped 1 previous similar message [ 5411.254959] Lustre: lustre-MDT0001: new disk, initializing [ 5411.292895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5411.307114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5411.310042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5412.648878] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5415.606018] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5416.796472] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5419.105824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5419.135085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5419.243046] Lustre: lustre-OST0000: new disk, initializing [ 5419.244833] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5419.275781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5421.039280] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5424.135327] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5425.479313] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5427.196865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5427.201240] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5427.231967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5427.626838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5428.941560] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5429.006270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5432.293133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5432.293522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5432.297064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5432.302420] Lustre: Skipped 1 previous similar message [ 5435.692232] LustreError: 385273:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5435.755524] Lustre: server umount lustre-OST0000 complete [ 5437.409233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5437.409762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5437.412213] Lustre: Skipped 1 previous similar message [ 5437.413596] Lustre: Skipped 1 previous similar message [ 5442.529046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5442.530620] Lustre: Skipped 2 previous similar messages [ 5443.131213] LustreError: 385475:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5443.132781] LustreError: 385475:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5443.160971] Lustre: server umount lustre-MDT0000 complete [ 5444.288659] LustreError: 384064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758829061 with bad export cookie 8307389547472510951 [ 5444.292847] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5444.294069] LustreError: 384064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5444.361972] LustreError: 385677:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5444.363629] LustreError: 385677:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5444.451501] Lustre: server umount lustre-MDT0001 complete [ 5447.643051] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5448.258995] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:37:45 (1758829065) [ 5450.673865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5450.915253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5452.704840] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5456.283028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5456.354177] LustreError: 386396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5456.368600] LustreError: 386396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5456.447192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5457.885453] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5459.981978] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5461.266798] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5463.678035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5466.413134] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5468.545662] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5471.637366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5471.686177] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5471.776484] Lustre: lustre-OST0001: new disk, initializing [ 5471.779168] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5471.826873] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5471.828979] Lustre: Skipped 1 previous similar message [ 5472.744601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5472.750225] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5472.797256] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5473.962267] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5477.788738] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5480.945652] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5480.949086] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5480.952538] Lustre: Skipped 1 previous similar message [ 5480.958750] Lustre: Skipped 1 previous similar message [ 5482.037178] LustreError: 389596:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5482.041365] LustreError: 389596:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5482.069540] Lustre: server umount lustre-OST0001 complete [ 5483.016342] LustreError: 389041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5486.063743] LustreError: 387990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5486.069034] LustreError: 387990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5491.178334] LustreError: 387990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5491.186084] LustreError: 387990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5496.297717] LustreError: 387991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5496.297844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5496.300797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5496.301426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5496.301437] LustreError: Skipped 1 previous similar message [ 5496.303869] LustreError: 387991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5496.310019] Lustre: Skipped 1 previous similar message [ 5500.659181] LustreError: 389818:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5500.663448] LustreError: 389818:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5500.771638] Lustre: server umount lustre-OST0000 complete [ 5506.531041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5506.532313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5506.537287] Lustre: Skipped 1 previous similar message [ 5506.539638] Lustre: Skipped 1 previous similar message [ 5508.868886] LustreError: 390019:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5508.870615] LustreError: 390019:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5508.900889] Lustre: server umount lustre-MDT0000 complete [ 5510.041442] LustreError: 386376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758829127 with bad export cookie 8307389547472511819 [ 5510.045971] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5516.251961] Lustre: server umount lustre-MDT0001 complete [ 5519.468326] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 5520.542745] Key type lgssc unregistered [ 5520.670537] LNet: 390752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5520.672683] LNetError: 390752:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5520.685398] LNet: Removed LNI 192.168.201.124@tcp [ 5520.956292] Key type .llcrypt unregistered [ 5520.957338] Key type ._llcrypt unregistered [ 5534.323771] Key type ._llcrypt registered [ 5534.324694] Key type .llcrypt registered [ 5534.357911] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 5540.647251] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5540.977975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5541.002054] alg: No test for adler32 (adler32-zlib) [ 5541.866120] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 5541.959616] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 5543.568125] Key type lgssc registered [ 5543.897202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5547.591486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5549.854172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5552.404219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5555.410571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5558.567389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5558.597161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5558.603918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5559.697965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5559.718302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5559.775960] Lustre: lustre-MDT0000: new disk, initializing [ 5559.806530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5559.819960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5561.543038] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5565.566719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5565.588510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5565.629278] Lustre: 394674:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5565.654720] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5565.656475] Lustre: Skipped 1 previous similar message [ 5565.692151] Lustre: lustre-MDT0001: new disk, initializing [ 5565.738982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5565.760181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5565.765053] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5566.939972] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5569.785639] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5571.488622] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5574.769762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5574.811917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5574.950113] Lustre: lustre-OST0000: new disk, initializing [ 5574.953773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5574.998586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5575.114946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5575.118498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5575.157428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5577.500304] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5581.480572] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5583.245929] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5583.339201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5585.022622] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5585.145415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5590.502826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5590.503124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5590.507143] LustreError: Skipped 1 previous similar message [ 5590.517440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5590.523521] Lustre: Skipped 1 previous similar message [ 5595.622729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5595.626294] Lustre: Skipped 1 previous similar message [ 5600.224207] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5600.248098] LustreError: 397609:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5600.330594] Lustre: server umount lustre-OST0000 complete [ 5605.859614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5605.860822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5605.867046] Lustre: Skipped 2 previous similar messages [ 5608.060286] LustreError: 397812:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5608.063249] LustreError: 397812:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5608.093313] Lustre: server umount lustre-MDT0000 complete [ 5609.234237] LustreError: 394665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758829226 with bad export cookie 16342321994887565869 [ 5609.237666] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5609.238807] LustreError: 394665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5609.299425] LustreError: 398013:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5609.301962] LustreError: 398013:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5609.377555] Lustre: server umount lustre-MDT0001 complete [ 5612.993791] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 15:40:30 (1758829230) [ 5616.055765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5619.575490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5619.735449] Lustre: lustre-MDT0000: new disk, initializing [ 5619.765754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5621.495960] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5624.802093] LustreError: 399280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5624.811299] LustreError: 399280:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5625.487443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5625.645428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5627.295499] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5629.612413] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5631.200477] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5635.042507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5635.043701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5635.049640] Lustre: Skipped 1 previous similar message [ 5635.058371] Lustre: Skipped 2 previous similar messages [ 5638.352659] LustreError: 400622:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5638.356064] LustreError: 400622:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5638.387810] Lustre: server umount lustre-MDT0000 complete [ 5639.936150] LustreError: 399259:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758829257 with bad export cookie 16342321994887566737 [ 5639.939163] LustreError: 399259:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5639.940305] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5640.084299] Lustre: server umount lustre-MDT0001 complete [ 5644.183917] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:41:01 (1758829261) [ 5646.788809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5647.042192] LustreError: 401546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5647.086747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5648.655505] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5652.130111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5654.152729] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5656.752725] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5658.307671] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5661.708875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5664.528891] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5666.760954] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5680.597842] Lustre: Permanently deactivating lustre-OST0000 [ 5680.609321] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5688.820862] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5689.793365] LustreError: 404307:0:(mgs_llog.c:1937:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5689.801030] LustreError: 404307:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5689.806517] LustreError: 404307:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5690.620966] LustreError: 404355:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5690.624610] LustreError: 404355:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5697.649121] LustreError: 404452:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5697.653677] LustreError: 404452:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5697.746744] Lustre: server umount lustre-OST0000 complete [ 5698.947548] LustreError: 404604:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5698.949350] LustreError: 404604:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5703.137603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5703.138261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5703.141211] Lustre: Skipped 1 previous similar message [ 5703.142720] Lustre: Skipped 2 previous similar messages [ 5710.308458] LustreError: 404701:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5710.311309] LustreError: 404701:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5710.333863] Lustre: server umount lustre-MDT0000 complete [ 5711.490247] LustreError: 401525:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758829329 with bad export cookie 16342321994887567276 [ 5711.493236] LustreError: 401525:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5711.494409] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5722.175264] Lustre: server umount lustre-MDT0001 complete [ 5724.897517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5726.166839] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5727.111864] LustreError: 405744:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5729.197178] Lustre: 406082:0:(mgs_llog.c:1698:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.124@tcp [ 5729.861657] LustreError: 406180:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5729.863391] LustreError: 406180:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5729.889556] Lustre: server umount MGS complete [ 5733.470409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5733.610349] LustreError: 406808:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5733.614702] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5733.616966] Lustre: Skipped 1 previous similar message [ 5733.633883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5733.635664] Lustre: Skipped 2 previous similar messages [ 5734.808742] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5737.528693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5737.623311] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5738.910657] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5740.651594] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5741.825725] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5742.443582] Lustre: Permanently reactivating lustre-OST0000 [ 5742.448162] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5742.449917] Lustre: Skipped 2 previous similar messages [ 5767.648198] LustreError: 408363:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5771.615026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5771.814275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5771.818270] Lustre: Skipped 1 previous similar message [ 5773.894479] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5775.614185] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5788.130705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5788.131368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5788.132468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5788.132473] Lustre: Skipped 3 previous similar messages [ 5788.138179] Lustre: Skipped 1 previous similar message [ 5789.486940] LustreError: 410191:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5789.488668] LustreError: 410191:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5789.549488] Lustre: server umount lustre-OST0000 complete [ 5792.783423] Lustre: server umount lustre-MDT0000 complete [ 5793.251418] LustreError: 406807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5793.255324] LustreError: 406807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5793.927078] LustreError: 406788:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758829411 with bad export cookie 16342321994887568221 [ 5793.929722] LustreError: 406788:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5793.930041] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5794.056317] Lustre: server umount lustre-MDT0001 complete [ 5797.736249] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 5798.770971] Key type lgssc unregistered [ 5798.897383] LNet: 411126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5798.899346] LNetError: 411126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5798.907324] LNet: Removed LNI 192.168.201.124@tcp [ 5799.206085] Key type .llcrypt unregistered [ 5799.207351] Key type ._llcrypt unregistered [ 5805.371185] Key type ._llcrypt registered [ 5805.372615] Key type .llcrypt registered [ 5805.412690] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 5812.516216] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5812.889585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5812.901900] alg: No test for adler32 (adler32-zlib) [ 5813.755633] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 5813.839952] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 5815.424207] Key type lgssc registered [ 5815.879969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5819.272384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5821.683408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5824.119511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5826.676655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5829.038326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5829.065683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5829.070875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5830.179257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5830.193001] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5830.236645] Lustre: lustre-MDT0000: new disk, initializing [ 5830.261528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5830.269393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5831.683439] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5836.804690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5836.838462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5836.885780] Lustre: 414230:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5836.920362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5836.924160] Lustre: Skipped 1 previous similar message [ 5836.972234] Lustre: lustre-MDT0001: new disk, initializing [ 5837.009401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5837.021805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5837.024641] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5838.697752] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5841.849358] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5843.010188] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5845.130213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5845.153879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5845.247866] Lustre: lustre-OST0000: new disk, initializing [ 5845.249504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5845.274193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5846.996117] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5849.990592] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5851.242902] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5853.689963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5853.692725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5853.727793] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5854.371242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5855.616201] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5855.693949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5858.786629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5858.788335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.789240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5858.790087] LustreError: Skipped 1 previous similar message [ 5858.795065] Lustre: Skipped 1 previous similar message [ 5862.381680] LustreError: 417187:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5862.453309] Lustre: server umount lustre-OST0000 complete [ 5863.905343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5863.905984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5863.908328] Lustre: Skipped 1 previous similar message [ 5863.909745] Lustre: Skipped 1 previous similar message [ 5869.025136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5869.026958] Lustre: Skipped 1 previous similar message [ 5869.755982] LustreError: 417388:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5869.757773] LustreError: 417388:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5869.785618] Lustre: server umount lustre-MDT0000 complete [ 5870.959035] LustreError: 414220:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758829488 with bad export cookie 8693311282190169856 [ 5870.962018] LustreError: 414220:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5870.962152] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5871.028791] LustreError: 417590:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5871.030580] LustreError: 417590:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5871.108074] Lustre: server umount lustre-MDT0001 complete [ 5873.798226] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5874.355591] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:44:51 (1758829491) [ 5876.471775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5876.636081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5877.891939] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5880.593640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5880.719689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5882.064651] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5883.809817] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5884.937914] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5887.415439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5889.423046] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5891.148305] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5891.838744] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5902.528946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5906.401940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5906.403808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5906.407159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5907.427223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5909.999143] LustreError: 420765:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5910.000803] LustreError: 420765:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5910.079273] Lustre: server umount lustre-OST0000 complete [ 5911.520987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5911.522845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5911.525522] Lustre: Skipped 1 previous similar message [ 5911.526994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5911.528438] Lustre: Skipped 2 previous similar messages [ 5917.439476] LustreError: 420967:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5917.442624] LustreError: 420967:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5917.476192] Lustre: server umount lustre-MDT0000 complete [ 5917.665381] LustreError: 418306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5917.669496] LustreError: 418306:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5918.694448] LustreError: 420679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758829536 with bad export cookie 8693311282190170738 [ 5918.698680] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5918.871815] Lustre: server umount lustre-MDT0001 complete [ 5922.423529] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 5923.495182] Key type lgssc unregistered [ 5923.631442] LNet: 421701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5923.634079] LNetError: 421701:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5923.642387] LNet: Removed LNI 192.168.201.124@tcp [ 5924.043455] Key type .llcrypt unregistered [ 5924.045482] Key type ._llcrypt unregistered [ 5932.316967] Key type ._llcrypt registered [ 5932.317990] Key type .llcrypt registered [ 5932.358339] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 5932.481347] Key type .llcrypt unregistered [ 5932.482741] Key type ._llcrypt unregistered [ 5939.665047] Key type ._llcrypt registered [ 5939.666623] Key type .llcrypt registered [ 5939.707232] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5940.058474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5940.067770] alg: No test for adler32 (adler32-zlib) [ 5940.951506] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 5941.045040] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 5942.624207] Key type lgssc registered [ 5942.996170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5945.678427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5945.685124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5946.849763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5948.276293] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5951.053495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5951.166136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5953.048512] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5954.816728] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5956.577767] LustreError: 423849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5956.582728] LustreError: 423849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5956.909627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5957.027562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5958.056518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5958.482901] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5962.209824] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5962.215468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5962.225498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5963.235343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5963.239609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5963.242670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5967.537079] LustreError: 425902:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5967.620670] Lustre: server umount lustre-OST0000 complete [ 5971.938299] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5971.940387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5971.943754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5971.946514] Lustre: Skipped 1 previous similar message [ 5974.907265] LustreError: 426103:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5974.908977] LustreError: 426103:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5974.932675] Lustre: server umount lustre-MDT0000 complete [ 5976.158312] LustreError: 423829:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758829593 with bad export cookie 3950278866336986445 [ 5976.163421] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5976.238254] LustreError: 426307:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5976.240337] LustreError: 426307:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5976.330821] Lustre: server umount lustre-MDT0001 complete [ 5980.188136] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 5981.274281] Key type lgssc unregistered [ 5981.406534] LNet: 426838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5981.409404] LNetError: 426838:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5981.420426] LNet: Removed LNI 192.168.201.124@tcp [ 5981.739127] Key type .llcrypt unregistered [ 5981.741174] Key type ._llcrypt unregistered [ 5991.278192] Key type ._llcrypt registered [ 5991.279399] Key type .llcrypt registered [ 5991.323657] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5991.732838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5991.778547] alg: No test for adler32 (adler32-zlib) [ 5992.632678] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 5992.721131] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 5994.304143] Key type lgssc registered [ 5994.633852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5997.400748] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5997.409560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6012.832330] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6017.952239] LustreError: 428184:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 6044.576174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6046.495104] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6048.345551] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6051.205757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6051.374833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6051.400919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6052.689253] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6055.828477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6055.971706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6057.300940] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6059.270481] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6060.528360] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6065.125398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6065.130090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6065.136177] Lustre: Skipped 1 previous similar message [ 6069.166127] LustreError: 430465:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 6069.239089] Lustre: server umount lustre-OST0000 complete [ 6071.265326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6071.269552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6071.276876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6071.282180] Lustre: Skipped 1 previous similar message [ 6075.362276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6075.362863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.738143] LustreError: 430668:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6076.741261] LustreError: 430668:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6076.764729] Lustre: server umount lustre-MDT0000 complete [ 6077.864116] LustreError: 428192:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758829695 with bad export cookie 3023892265964938103 [ 6077.867094] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6077.869281] LustreError: 428192:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6077.928762] LustreError: 430871:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6077.931226] LustreError: 430871:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6078.006972] Lustre: server umount lustre-MDT0001 complete [ 6081.413368] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6082.357330] Key type lgssc unregistered [ 6082.478316] LNet: 431402:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6082.480245] LNetError: 431402:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6082.486282] LNet: Removed LNI 192.168.201.124@tcp [ 6082.776433] Key type .llcrypt unregistered [ 6082.777334] Key type ._llcrypt unregistered [ 6091.340492] Key type ._llcrypt registered [ 6091.341970] Key type .llcrypt registered [ 6091.374948] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6091.650188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6091.659731] alg: No test for adler32 (adler32-zlib) [ 6092.497529] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 6092.579995] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 6094.160113] Key type lgssc registered [ 6094.506829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6096.886333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6096.892572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6098.037407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6099.280195] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6101.920286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6102.032172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6103.167481] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6104.786606] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6105.903547] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6107.974651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6108.077796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6109.095931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6109.743092] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6111.365237] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6117.348578] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6117.349899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6117.350406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6117.353859] Lustre: Skipped 2 previous similar messages [ 6118.011209] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.24@tcp (stopping) [ 6118.012810] Lustre: Skipped 2 previous similar messages [ 6119.294734] LustreError: 435017:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6119.326924] Lustre: server umount lustre-MDT0000 complete [ 6120.052168] LustreError: 432755:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.24@tcp arrived at 1758829737 with bad export cookie 10809966653712349177 [ 6120.055903] LustreError: 432755:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6120.663119] LustreError: 432755:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758829738 with bad export cookie 10809966653712348330 [ 6120.666370] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6120.670466] LustreError: 432755:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6122.465329] LustreError: 432775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6122.465691] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6122.465989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6122.466409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6122.469192] LustreError: 432775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6125.472941] LustreError: 432775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6125.473277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6125.478139] LustreError: 432775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6126.765094] LustreError: 435221:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6126.766763] LustreError: 435221:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6126.823359] Lustre: server umount lustre-OST0000 complete [ 6128.862365] LustreError: 435575:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6128.864014] LustreError: 435575:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6128.938557] Lustre: server umount lustre-MDT0001 complete [ 6132.436217] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6133.407297] Key type lgssc unregistered [ 6133.532400] LNet: 436107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6133.534234] LNetError: 436107:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6133.543293] LNet: Removed LNI 192.168.201.124@tcp [ 6133.836765] Key type .llcrypt unregistered [ 6133.837644] Key type ._llcrypt unregistered [ 6142.424772] Key type ._llcrypt registered [ 6142.425861] Key type .llcrypt registered [ 6142.459385] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6142.771237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6142.804579] alg: No test for adler32 (adler32-zlib) [ 6143.651591] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 6143.725574] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 6145.296104] Key type lgssc registered [ 6145.599364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6147.885475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6147.893177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6149.053099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6150.336402] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6153.223199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6153.351886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6154.573143] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6156.348586] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6157.523888] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6159.718340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6159.832507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6161.596849] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6163.309581] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6174.188640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6184.059548] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.24@tcp (stopping) [ 6184.419678] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6184.419721] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6189.173449] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.24@tcp (stopping) [ 6189.175803] Lustre: Skipped 2 previous similar messages [ 6194.144106] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6194.180157] LustreError: 439744:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6194.195958] Lustre: server umount lustre-MDT0001 complete [ 6194.293525] LustreError: 437474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6194.297756] LustreError: 437474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6199.777396] LustreError: 437479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6199.777901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6199.778599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6199.778605] Lustre: Skipped 1 previous similar message [ 6199.779098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6199.779101] Lustre: Skipped 2 previous similar messages [ 6199.783753] LustreError: 437479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6201.834791] LustreError: 439960:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6201.836680] LustreError: 439960:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6201.902210] Lustre: server umount lustre-OST0000 complete [ 6203.141328] LustreError: 440161:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6203.144109] LustreError: 440161:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6203.226287] Lustre: server umount lustre-MDT0000 complete [ 6207.608688] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6208.624121] Key type lgssc unregistered [ 6208.778534] LNet: 440845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6208.782350] LNetError: 440845:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6208.792467] LNet: Removed LNI 192.168.201.124@tcp [ 6209.102389] Key type .llcrypt unregistered [ 6209.103306] Key type ._llcrypt unregistered [ 6216.979766] Key type ._llcrypt registered [ 6216.980826] Key type .llcrypt registered [ 6217.012124] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 6222.911153] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6223.302614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6223.312570] alg: No test for adler32 (adler32-zlib) [ 6224.152641] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 6224.231844] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 6225.808132] Key type lgssc registered [ 6226.174267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6229.100469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6231.254623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6233.261151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6235.292401] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6238.295402] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6241.101746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6243.303801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6243.323155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6243.327644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6244.433832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6244.447414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6244.491388] Lustre: lustre-MDT0000: new disk, initializing [ 6244.516130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6244.522713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6245.843701] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6249.285467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6251.533134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6251.557809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6251.646876] Lustre: lustre-OST0000: new disk, initializing [ 6251.648664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6251.650174] Lustre: Skipped 1 previous similar message [ 6251.672326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6253.696226] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6254.028481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6254.030945] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6254.043452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6256.964673] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6260.206921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6262.477568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6262.510770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6262.551053] Lustre: 444790:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6262.586644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6262.632500] Lustre: lustre-MDT0001: new disk, initializing [ 6262.684788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6262.702759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6262.707953] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6264.050609] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6267.519597] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6267.589189] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6268.828367] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6268.898437] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6272.131676] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.24@tcp (stopping) [ 6274.531919] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6274.532099] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6274.532900] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6274.534713] Lustre: Skipped 1 previous similar message [ 6276.476330] LustreError: 448208:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6276.500557] Lustre: server umount lustre-MDT0001 complete [ 6284.771470] LustreError: 447012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6284.776048] LustreError: 447012:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6289.889781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6289.890264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6289.890676] LustreError: 447012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6289.898474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6289.899820] Lustre: Skipped 1 previous similar message [ 6295.009341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6295.009994] LustreError: 447012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6300.129341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6300.130045] LustreError: 447012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6303.200197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6303.225188] LustreError: 448479:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6303.227504] LustreError: 448479:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6303.298682] Lustre: server umount lustre-OST0000 complete [ 6310.717179] LustreError: 448681:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6310.720305] LustreError: 448681:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6310.807582] Lustre: server umount lustre-MDT0000 complete [ 6314.857590] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6315.836090] Key type lgssc unregistered [ 6315.959491] LNet: 449366:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6315.963340] LNetError: 449366:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6315.979327] LNet: Removed LNI 192.168.201.124@tcp [ 6316.292228] Key type .llcrypt unregistered [ 6316.293825] Key type ._llcrypt unregistered [ 6327.922081] Key type ._llcrypt registered [ 6327.922948] Key type .llcrypt registered [ 6327.955331] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6328.053247] Key type .llcrypt unregistered [ 6328.054226] Key type ._llcrypt unregistered [ 6334.319204] Key type ._llcrypt registered [ 6334.320437] Key type .llcrypt registered [ 6334.356986] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6334.639699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6334.659455] alg: No test for adler32 (adler32-zlib) [ 6335.495655] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 6335.570454] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 6337.144118] Key type lgssc registered [ 6337.455857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6339.822563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6341.817641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6343.662494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6345.734513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6345.753056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6345.758076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6346.833869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6346.844280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6346.889181] Lustre: lustre-MDT0000: new disk, initializing [ 6346.913493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6346.921239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6348.070159] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6351.819457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6351.840983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6351.870242] Lustre: 452684:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6351.889873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6351.891796] Lustre: Skipped 1 previous similar message [ 6351.926272] Lustre: lustre-MDT0001: new disk, initializing [ 6351.956864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6351.966956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6351.969508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6353.035430] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6355.856667] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6356.891902] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6358.894689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6358.915605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6358.998350] Lustre: lustre-OST0000: new disk, initializing [ 6359.000092] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6359.023475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6360.635736] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6363.002205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6363.005076] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6363.041700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6363.402332] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6367.201063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6367.204343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6367.207609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6368.225662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6368.226781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6368.230114] Lustre: Skipped 1 previous similar message [ 6371.713348] LustreError: 455125:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6371.743153] Lustre: server umount lustre-MDT0000 complete [ 6372.912214] LustreError: 452675:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758829990 with bad export cookie 6390677105938841517 [ 6372.914987] LustreError: 452675:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6372.916175] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6372.973321] LustreError: 455327:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6372.975283] LustreError: 455327:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6373.054691] Lustre: server umount lustre-MDT0001 complete [ 6384.368289] LustreError: 455527:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6384.369864] LustreError: 455527:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6384.437382] Lustre: server umount lustre-OST0000 complete [ 6387.539531] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 15:53:25 (1758830005) [ 6387.852582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6389.852300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6390.008809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6391.136805] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6393.640967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6394.848547] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6396.486532] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6397.599373] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6399.839539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6399.866540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6399.945188] Lustre: Found index 0 for lustre-OST0000, updating log [ 6399.972046] LustreError: 457956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6399.979701] LustreError: 457956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6400.011733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6400.014235] Lustre: Skipped 1 previous similar message [ 6402.176266] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6404.137474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6404.356260] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6409.190589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6409.193579] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6409.206796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6409.206805] Lustre: Skipped 2 previous similar messages [ 6409.226838] LustreError: Skipped 1 previous similar message [ 6412.270959] LustreError: 458607:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6412.273519] LustreError: 458607:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6412.372767] Lustre: server umount lustre-OST0000 complete [ 6414.154644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6416.356199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6416.359313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6416.363375] Lustre: Skipped 2 previous similar messages [ 6416.369166] Lustre: Skipped 4 previous similar messages [ 6421.476495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6421.483915] Lustre: Skipped 1 previous similar message [ 6421.545945] LustreError: 458866:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6421.552352] LustreError: 458866:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6421.600870] Lustre: server umount lustre-MDT0000 complete [ 6424.220450] LustreError: 456335:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758830041 with bad export cookie 6390677105938842896 [ 6424.229828] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6424.238980] LustreError: 456335:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6424.483161] Lustre: server umount lustre-MDT0001 complete [ 6429.111721] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 15:54:06 (1758830046) [ 6429.545891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6429.917556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6430.258747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6430.589016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6430.909697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6431.232439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6431.535829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6431.847715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6432.155585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6432.480515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6432.790150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6433.101353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6433.456197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6433.786059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6434.091380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6434.387586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6434.684194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6435.008849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6435.304815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6435.626387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6435.945644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6436.268865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6436.578039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6436.914057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6437.227842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6437.547428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6437.859063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6438.191130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6438.507034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6438.810430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6439.127630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6439.446164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6439.758377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6440.091806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6440.446174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6440.765566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6441.098103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6441.416028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6441.737070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6442.060644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6442.367241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6442.666393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6442.992174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6443.299856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6443.609461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6443.948382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6444.269130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6444.563898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6444.861522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6445.170344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6445.477651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6447.891151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6448.061702] LustreError: 462448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6448.084894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6449.349861] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6451.982895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6453.374722] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6455.217357] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6456.456587] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6458.880835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6458.911933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6459.002990] Lustre: Found index 0 for lustre-OST0000, updating log [ 6460.140655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6461.249808] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6463.252876] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6465.504946] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6465.507128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6465.510854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6470.062312] LustreError: 464731:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6470.064291] LustreError: 464731:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6470.132720] Lustre: server umount lustre-OST0000 complete [ 6477.664134] Lustre: server umount lustre-MDT0000 complete [ 6477.665901] LustreError: 462448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6477.673780] LustreError: 462448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6479.009386] LustreError: 464112:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758830096 with bad export cookie 6390677105938843967 [ 6479.013387] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6479.017607] LustreError: 464112:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6479.192376] Lustre: server umount lustre-MDT0001 complete [ 6480.089147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6485.037097] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 6487.922228] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6491.101371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6493.197361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6495.163285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6497.170768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6499.219796] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 15:55:16 (1758830116) [ 6538.998554] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6540.092965] Key type lgssc unregistered [ 6540.237775] LNet: 468726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6540.242284] LNetError: 468726:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6670.311811] LNet: Removed LNI 192.168.201.124@tcp [ 6670.646238] Key type .llcrypt unregistered [ 6670.647155] Key type ._llcrypt unregistered [ 6677.998217] Key type ._llcrypt registered [ 6677.999620] Key type .llcrypt registered [ 6678.047620] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6678.398467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6678.439576] alg: No test for adler32 (adler32-zlib) [ 6679.295921] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 6679.393916] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 6680.984231] Key type lgssc registered [ 6681.391526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6683.231513] LNet: Added LNI 192.168.201.124@tcp100 [8/256/0/180] [ 6683.891331] Lustre: DEBUG MARKER: SKIP: conf-sanity test_73c unable to configure net [ 6690.989859] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6692.236731] Key type lgssc unregistered [ 6692.423699] LNet: 470595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6692.428065] LNetError: 470595:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6692.431239] LNet: Removed LNI 192.168.201.124@tcp [ 6692.442620] LNet: Removed LNI 192.168.201.124@tcp100 [ 6692.773752] Key type .llcrypt unregistered [ 6692.775448] Key type ._llcrypt unregistered [ 6707.785242] Key type ._llcrypt registered [ 6707.786660] Key type .llcrypt registered [ 6707.868482] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 6724.475284] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6725.932885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6725.975721] alg: No test for adler32 (adler32-zlib) [ 6727.085861] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 6727.313160] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 6729.088186] Key type lgssc registered [ 6730.343674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6740.917105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6748.671263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6754.521789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6763.255466] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6775.246966] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 15:59:51 (1758830391) [ 6799.162715] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6803.679515] Key type lgssc unregistered [ 6804.154093] LNet: 474638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6804.171895] LNetError: 474638:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6804.192697] LNet: Removed LNI 192.168.201.124@tcp [ 6805.540602] Key type .llcrypt unregistered [ 6805.542292] Key type ._llcrypt unregistered [ 6826.961393] Key type ._llcrypt registered [ 6826.963855] Key type .llcrypt registered [ 6827.112591] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6828.205509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6828.321901] alg: No test for adler32 (adler32-zlib) [ 6829.441565] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 6829.635921] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 6831.345869] Key type lgssc registered [ 6832.588755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6841.244176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6841.291813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6841.311918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6842.638162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6842.705882] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6842.792403] Lustre: lustre-MDT0000: new disk, initializing [ 6842.902181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6842.926382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6847.092290] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6856.578864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6856.656873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6856.758258] Lustre: 475831:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6856.823193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6856.828319] Lustre: Skipped 1 previous similar message [ 6856.884361] Lustre: lustre-MDT0001: new disk, initializing [ 6856.988485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6857.048228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6857.054989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6860.565987] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6867.335343] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6870.717152] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6876.582837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6876.644262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6876.870475] Lustre: lustre-OST0000: new disk, initializing [ 6876.878646] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6876.953207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6882.163175] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6883.421380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6883.443615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6883.583783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6889.772890] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6893.411108] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing lnet_if_list [ 6898.358181] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 6898.409749] LNet: Added LNI 192.168.201.124@tcp42 [8/256/0/180] [ 6914.530923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6914.531121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6914.545761] LustreError: Skipped 1 previous similar message [ 6914.554342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6919.364110] LustreError: 479225:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6919.545448] Lustre: server umount lustre-OST0000 complete [ 6923.745843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6923.748454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6923.765210] Lustre: Skipped 1 previous similar message [ 6923.768029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6923.780358] Lustre: Skipped 1 previous similar message [ 6924.769877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6924.770695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6929.151060] LustreError: 479426:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6929.173753] LustreError: 479426:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6929.328447] Lustre: server umount lustre-MDT0000 complete [ 6933.126217] LustreError: 475825:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758830550 with bad export cookie 2960295807105959140 [ 6933.133282] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6933.283659] LustreError: 479627:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6933.289297] LustreError: 479627:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6933.560058] Lustre: server umount lustre-MDT0001 complete [ 6943.629041] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6946.490452] Key type lgssc unregistered [ 6946.836397] LNet: 480167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6946.852584] LNetError: 480167:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6946.865555] LNet: Removed LNI 192.168.201.124@tcp [ 6946.881128] LNet: Removed LNI 192.168.201.124@tcp42 [ 6947.617244] Key type .llcrypt unregistered [ 6947.618730] Key type ._llcrypt unregistered [ 6964.593282] Key type ._llcrypt registered [ 6964.598289] Key type .llcrypt registered [ 6964.715565] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6965.579215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6965.790062] alg: No test for adler32 (adler32-zlib) [ 6966.948204] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 6967.170477] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 6968.816188] Key type lgssc registered [ 6970.007626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6976.921282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6976.957172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6978.448456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6981.934876] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6988.772677] LustreError: 481411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6988.784676] LustreError: 481411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6988.979543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6989.418235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6992.934932] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6998.111370] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7001.793880] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7008.139227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7008.545726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7014.289550] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 7020.148866] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7033.746636] loop0: detected capacity change from 0 to 204800000 [ 7033.874624] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7033.883902] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7033.898775] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7033.906063] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7033.946347] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7033.983946] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7034.016373] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7034.056382] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7034.095655] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7034.147435] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7039.376113] loop0: detected capacity change from 0 to 204800000 [ 7039.514859] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7039.526150] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7039.534409] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7039.652470] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7043.802374] loop0: detected capacity change from 0 to 204800000 [ 7048.653831] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7048.717279] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7048.822442] Lustre: fs15246-OST0000: new disk, initializing [ 7048.830518] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7048.887197] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7053.590500] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 7061.781460] loop1: detected capacity change from 0 to 204800000 [ 7066.354292] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7066.410587] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7066.476755] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7066.521484] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7066.575901] Lustre: fs15246-MDT0000: new disk, initializing [ 7066.693446] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7066.714315] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7066.800950] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7067.811619] Lustre: 485788:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000006b39f542 x1844267538489856/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1758830701 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7069.744508] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 7076.898048] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7076.905875] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7076.947327] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7089.131330] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7089.141415] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7090.852993] LustreError: 486913:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 7090.936946] Lustre: server umount fs15246-MDT0000 complete [ 7094.261897] LustreError: 486914:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7097.951109] LustreError: 487414:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 7097.957983] LustreError: 487414:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7097.990657] Lustre: server umount fs15246-OST0000 complete [ 7109.610933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7109.628631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7112.779080] LustreError: 487916:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7112.789068] LustreError: 487916:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7112.976778] Lustre: server umount lustre-OST0000 complete [ 7117.284293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7117.299593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7117.329776] Lustre: Skipped 1 previous similar message [ 7117.333071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7117.336192] Lustre: Skipped 1 previous similar message [ 7119.852144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7119.855277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7119.875149] Lustre: Skipped 1 previous similar message [ 7122.423434] LustreError: 488118:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7122.426307] LustreError: 488118:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7122.516625] Lustre: server umount lustre-MDT0000 complete [ 7124.969887] LustreError: 481412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7124.989829] LustreError: 481412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7125.823413] LustreError: 481392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758830743 with bad export cookie 16718438586705181631 [ 7125.832996] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7126.239844] Lustre: server umount lustre-MDT0001 complete [ 7136.278826] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 7139.340902] Key type lgssc unregistered [ 7139.705272] LNet: 488853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7139.716952] LNetError: 488853:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7140.773896] LNet: Removed LNI 192.168.201.124@tcp [ 7141.623962] Key type .llcrypt unregistered [ 7141.626732] Key type ._llcrypt unregistered [ 7159.786248] Key type ._llcrypt registered [ 7159.788480] Key type .llcrypt registered [ 7159.867881] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 7161.024890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7161.071846] alg: No test for adler32 (adler32-zlib) [ 7162.186312] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 7162.411189] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 7164.080217] Key type lgssc registered [ 7165.319741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7172.458770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7177.174124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7182.298026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7186.759069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7202.121402] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 7212.132478] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 7221.364571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7227.675188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7233.141249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7239.405874] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7246.074760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7246.181121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7246.221095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7247.443904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7247.481833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7247.578887] Lustre: lustre-MDT0000: new disk, initializing [ 7247.664973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7247.680178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7251.290360] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 7261.218675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7261.285412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7261.347957] Lustre: 493286:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7261.412330] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7261.414813] Lustre: Skipped 1 previous similar message [ 7261.506980] Lustre: lustre-MDT0001: new disk, initializing [ 7261.607143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7261.672658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7261.694933] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7265.293077] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 7275.095960] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7280.818265] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7292.719744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7292.944870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7293.744742] Lustre: lustre-OST0000: new disk, initializing [ 7293.753253] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7294.020739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7303.831549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7303.835253] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7304.115319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7306.665851] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 7317.963890] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7323.228062] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7323.460592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7328.351754] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7328.664110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7334.881295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7334.887380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7334.890301] LustreError: Skipped 1 previous similar message [ 7334.903904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7334.910837] Lustre: Skipped 1 previous similar message [ 7334.928947] Lustre: Skipped 1 previous similar message [ 7337.098282] LustreError: 496232:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7337.426933] Lustre: server umount lustre-OST0000 complete [ 7343.585579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7343.588507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7343.610414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7345.121306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7345.122052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7347.836535] LustreError: 496434:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7347.857809] LustreError: 496434:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7348.017555] Lustre: server umount lustre-MDT0000 complete [ 7352.708102] LustreError: 493280:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758830970 with bad export cookie 4345828917876804007 [ 7352.712737] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7352.759837] LustreError: 493280:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7353.022753] LustreError: 496637:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7353.039692] LustreError: 496637:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7353.320861] Lustre: server umount lustre-MDT0001 complete [ 7384.723569] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 7395.128933] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 7409.267413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7419.051527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7426.828237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7434.321889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7436.245025] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7293 sec ========== 16:10:52 (1758831052) [ 7437.908851] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:10:54 (1758831054) === [ 7441.487248] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:10:58 (1758831058) === [ 7484.731587] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 7488.015160] Key type lgssc unregistered [ 7488.395710] LNet: 501505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7488.406678] LNetError: 501505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7488.437646] LNet: Removed LNI 192.168.201.124@tcp [ 7489.416498] Key type .llcrypt unregistered [ 7489.422274] Key type ._llcrypt unregistered