[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 560673476 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003320] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005009] kvm-guest: setup PV IPIs [ 0.008561] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010008] pid_max: default: 32768 minimum: 301 [ 0.011121] LSM: Security Framework initializing [ 0.012034] Yama: becoming mindful. [ 0.013024] SELinux: Initializing. [ 0.014050] *** VALIDATE selinux *** [ 0.022461] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026747] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027133] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028099] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029097] *** VALIDATE tmpfs *** [ 0.031178] *** VALIDATE proc *** [ 0.032195] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.035252] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038023] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042552] debug: unmapping init [mem 0xffffffffaac59000-0xffffffffaac60fff] [ 0.044296] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045651] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046019] ... version: 2 [ 0.047009] ... bit width: 48 [ 0.048009] ... generic registers: 4 [ 0.049008] ... value mask: 0000ffffffffffff [ 0.050009] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052008] ... event mask: 000000070000000f [ 0.054264] rcu: Hierarchical SRCU implementation. [ 0.056376] smp: Bringing up secondary CPUs ... [ 0.057523] x86: Booting SMP configuration: [ 0.058026] .... node #0, CPUs: #1 #2 #3 [ 0.067181] smp: Brought up 1 node, 4 CPUs [ 0.069010] smpboot: Max logical packages: 1 [ 0.070008] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.102464] node 0 deferred pages initialised in 30ms [ 0.106159] devtmpfs: initialized [ 0.107329] x86/mm: Memory block size: 128MB [ 0.109954] gcov: version magic: 0x41383552 [ 0.113334] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.114064] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.117247] pinctrl core: initialized pinctrl subsystem [ 0.119148] [ 0.119769] ************************************************************* [ 0.123012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.125011] ** ** [ 0.128012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.130009] ** ** [ 0.133009] ** This means that this kernel is built to expose internal ** [ 0.135007] ** IOMMU data structures, which may compromise security on ** [ 0.138011] ** your system. ** [ 0.140009] ** ** [ 0.143008] ** If you see this message and you are not debugging the ** [ 0.145008] ** kernel, report this immediately to your vendor! ** [ 0.148009] ** ** [ 0.150009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153009] ************************************************************* [ 0.155833] NET: Registered protocol family 16 [ 0.158411] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.161046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.164053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.167693] cpuidle: using governor menu [ 0.170009] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.172495] PCI: Using configuration type 1 for base access [ 0.175203] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.183309] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.184013] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.188101] cryptd: max_cpu_qlen set to 1000 [ 0.189258] ACPI: Added _OSI(Module Device) [ 0.191010] ACPI: Added _OSI(Processor Device) [ 0.192070] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.194011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.199979] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.207224] ACPI: Interpreter enabled [ 0.208044] ACPI: PM: (supports S0 S3 S4 S5) [ 0.210009] ACPI: Using IOAPIC for interrupt routing [ 0.212079] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.215532] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.227906] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.230027] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.233015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.237070] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.242732] acpiphp: Slot [2] registered [ 0.244104] acpiphp: Slot [5] registered [ 0.246121] acpiphp: Slot [6] registered [ 0.248103] acpiphp: Slot [7] registered [ 0.249098] acpiphp: Slot [8] registered [ 0.251163] acpiphp: Slot [9] registered [ 0.252085] acpiphp: Slot [10] registered [ 0.253000] acpiphp: Slot [3] registered [ 0.253000] acpiphp: Slot [4] registered [ 0.255110] acpiphp: Slot [11] registered [ 0.257087] acpiphp: Slot [12] registered [ 0.259132] acpiphp: Slot [13] registered [ 0.261109] acpiphp: Slot [14] registered [ 0.264092] acpiphp: Slot [15] registered [ 0.266086] acpiphp: Slot [16] registered [ 0.269087] acpiphp: Slot [17] registered [ 0.271104] acpiphp: Slot [18] registered [ 0.273071] acpiphp: Slot [19] registered [ 0.275067] acpiphp: Slot [20] registered [ 0.276170] acpiphp: Slot [21] registered [ 0.278070] acpiphp: Slot [22] registered [ 0.280414] acpiphp: Slot [23] registered [ 0.282068] acpiphp: Slot [24] registered [ 0.284079] acpiphp: Slot [25] registered [ 0.286092] acpiphp: Slot [26] registered [ 0.288306] acpiphp: Slot [27] registered [ 0.290100] acpiphp: Slot [28] registered [ 0.293092] acpiphp: Slot [29] registered [ 0.295157] acpiphp: Slot [30] registered [ 0.299098] acpiphp: Slot [31] registered [ 0.302085] PCI host bridge to bus 0000:00 [ 0.304032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.308023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.310019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.312020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.315022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.323022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.331166] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.340512] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.344481] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.356013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.361000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.363064] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.366014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.379016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.505716] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.507837] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.508000] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.508000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.508000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.513014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.519012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.525511] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.534014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.543014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.559014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.566000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.566000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.572000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.584014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.687000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.729013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.823038] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.834036] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.852732] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.009000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.013016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.030016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.049272] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.058014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.064082] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.077016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.090550] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.099014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.106016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.124015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.143385] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.147340] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.152236] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.159090] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.162215] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.176065] iommu: Default domain type: Passthrough [ 1.179441] SCSI subsystem initialized [ 1.183164] ACPI: bus type USB registered [ 1.186123] usbcore: registered new interface driver usbfs [ 1.189083] usbcore: registered new interface driver hub [ 1.195106] usbcore: registered new device driver usb [ 1.204188] pps_core: LinuxPPS API ver. 1 registered [ 1.207011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.214040] PTP clock support registered [ 1.218323] EDAC MC: Ver: 3.0.0 [ 1.219411] PCI: Using ACPI for IRQ routing [ 1.220000] NetLabel: Initializing [ 1.221009] NetLabel: domain hash size = 128 [ 1.223008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.225069] NetLabel: unlabeled traffic allowed by default [ 1.227234] vgaarb: loaded [ 1.228383] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.230012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.238549] clocksource: Switched to clocksource kvm-clock [ 1.577716] VFS: Disk quotas dquot_6.6.0 [ 1.580030] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.585031] *** VALIDATE ramfs *** [ 1.586413] *** VALIDATE hugetlbfs *** [ 1.588360] pnp: PnP ACPI init [ 1.591418] pnp: PnP ACPI: found 6 devices [ 1.679799] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.686875] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.691967] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.696620] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.701313] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.707621] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.713779] NET: Registered protocol family 2 [ 1.717291] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.729804] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.736746] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.748481] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.756456] TCP: Hash tables configured (established 65536 bind 65536) [ 1.763450] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.772918] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.780732] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.787124] NET: Registered protocol family 1 [ 1.792501] RPC: Registered named UNIX socket transport module. [ 1.798678] RPC: Registered udp transport module. [ 1.803354] RPC: Registered tcp transport module. [ 1.805739] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.808710] NET: Registered protocol family 44 [ 1.812618] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.815476] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.819699] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.824098] PCI: CLS 0 bytes, default 64 [ 1.827827] Unpacking initramfs... [ 4.750162] debug: unmapping init [mem 0xffff9c367cc54000-0xffff9c367ffbffff] [ 4.753569] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.755232] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.757464] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.262655] Initialise system trusted keyrings [ 5.264336] Key type blacklist registered [ 5.267096] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.276028] zbud: loaded [ 5.278735] *** VALIDATE nfs *** [ 5.280064] *** VALIDATE nfs4 *** [ 5.281502] pstore: using deflate compression [ 5.285688] Platform Keyring initialized [ 5.423242] NET: Registered protocol family 38 [ 5.426463] Key type asymmetric registered [ 5.427765] Asymmetric key parser 'x509' registered [ 5.430281] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.435161] io scheduler mq-deadline registered [ 5.437451] io scheduler kyber registered [ 5.439765] io scheduler bfq registered [ 5.443374] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.447531] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.453332] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.459550] ACPI: Power Button [PWRF] [ 5.466916] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.478185] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.660691] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.668614] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.684700] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.710451] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.738498] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.743405] Non-volatile memory driver v1.3 [ 5.745363] Linux agpgart interface v0.103 [ 5.776206] virtio_blk virtio1: [vda] 134224 512-byte logical blocks (68.7 MB/65.5 MiB) [ 5.779085] vda: detected capacity change from 0 to 68722688 [ 5.793612] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.797030] vdb: detected capacity change from 0 to 1073741824 [ 5.811301] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.814710] vdc: detected capacity change from 0 to 2621440000 [ 5.830536] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.834244] vdd: detected capacity change from 0 to 2621440000 [ 5.849674] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.852039] vde: detected capacity change from 0 to 4294967296 [ 5.869640] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.872928] vdf: detected capacity change from 0 to 4294967296 [ 5.879692] libphy: Fixed MDIO Bus: probed [ 5.885731] usbcore: registered new interface driver usbserial_generic [ 5.888451] usbserial: USB Serial support registered for generic [ 5.891148] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.895902] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.897749] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.900406] mousedev: PS/2 mouse device common for all mice [ 5.904067] rtc_cmos 00:05: RTC can wake from S4 [ 5.908218] rtc_cmos 00:05: registered as rtc0 [ 5.909874] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.911935] intel_pstate: CPU model not supported [ 5.922562] hid: raw HID events driver (C) Jiri Kosina [ 5.925877] usbcore: registered new interface driver usbhid [ 5.928795] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.937410] usbhid: USB HID core driver [ 5.940725] drop_monitor: Initializing network drop monitor service [ 5.949612] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.959229] Initializing XFRM netlink socket [ 5.968230] NET: Registered protocol family 10 [ 5.974534] Segment Routing with IPv6 [ 5.979548] NET: Registered protocol family 17 [ 5.982596] mpls_gso: MPLS GSO support [ 5.988352] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.004300] RAS: Correctable Errors collector initialized. [ 6.010595] AVX version of gcm_enc/dec engaged. [ 6.014266] AES CTR mode by8 optimization enabled [ 6.230648] sched_clock: Marking stable (6230631236, 0)->(7446999011, -1216367775) [ 6.239405] registered taskstats version 1 [ 6.243654] Loading compiled-in X.509 certificates [ 6.247637] zswap: loaded using pool lzo/zbud [ 6.362769] Key type big_key registered [ 6.414242] Key type encrypted registered [ 6.421898] ima: No TPM chip found, activating TPM-bypass! [ 6.430544] ima: Allocated hash algorithm: sha1 [ 6.439428] ima: No architecture policies found [ 6.447160] evm: Initialising EVM extended attributes: [ 6.455322] evm: security.selinux [ 6.459654] evm: security.ima [ 6.464414] evm: security.capability [ 6.472166] evm: HMAC attrs: 0x1 [ 6.487604] rtc_cmos 00:05: setting system clock to 2026-02-07 06:45:20 UTC (1770446720) [ 6.523430] debug: unmapping init [mem 0xffffffffabc03000-0xffffffffabdfffff] [ 6.536238] debug: unmapping init [mem 0xffffffffaa982000-0xffffffffaac58fff] [ 6.550022] Write protecting the kernel read-only data: 28672k [ 6.559727] debug: unmapping init [mem 0xffffffffa9003000-0xffffffffa91fffff] [ 6.570542] debug: unmapping init [mem 0xffffffffa9914000-0xffffffffa99fffff] [ 6.622676] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.634701] systemd[1]: Detected virtualization kvm. [ 6.638362] systemd[1]: Detected architecture x86-64. [ 6.641349] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.670697] systemd[1]: No hostname configured. [ 6.673271] systemd[1]: Set hostname to . [ 6.675760] random: systemd: uninitialized urandom read (16 bytes read) [ 6.680197] systemd[1]: Initializing machine ID from random generator. [ 6.785166] random: ln: uninitialized urandom read (6 bytes read) [ 6.979437] random: systemd: uninitialized urandom read (16 bytes read) [ 6.981908] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 6.989596] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 6.998871] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.172392] device-mapper: uevent: version 1.0.3 [ 9.174392] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 11.208358] virtio_net virtio0 ens2: renamed from eth0 [ 11.237867] random: fast init done [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 11.857411] scsi host0: ata_piix [ 11.913020] scsi host1: ata_piix [ 11.939203] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.954807] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.993620] random: crng init done [ 17.996891] random: 7 urandom warning(s) missed due to ratelimiting [ 20.527464] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 22.371189] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.725371] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.077047] SELinux: Disabled at runtime. [ 26.323850] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.331308] systemd[1]: Detected virtualization kvm. [ 26.333491] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.195577] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.199537] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.210869] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.216128] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.219557] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.228492] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.237927] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ 28.591494] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Mounted /mnt. [ 30.036520] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.666571] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 30.714905] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 33.411282] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 33.527097] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 36.731272] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 37.365892] NFS: Registering the id_resolver key type [ 37.370175] Key type id_resolver registered [ 37.371538] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg316-server login: [ 72.039092] libcfs: loading out-of-tree module taints kernel. [ 72.076496] Key type ._llcrypt registered [ 72.078320] Key type .llcrypt registered [ 72.157489] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 80.876332] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 81.625971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 81.634465] alg: No test for adler32 (adler32-zlib) [ 82.650332] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 82.944480] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 84.551961] Key type lgssc registered [ 85.450364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.440726] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 115.129459] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 122.011882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 122.032865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 123.205996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 123.223898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 123.272703] Lustre: lustre-MDT0000: new disk, initializing [ 123.334596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 123.343486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 125.058765] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 131.850269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 131.893731] Lustre: 6474:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 131.908089] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 131.912520] Lustre: Skipped 1 previous similar message [ 131.961443] Lustre: lustre-MDT0001: new disk, initializing [ 131.989902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 132.002289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 132.005420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 133.801121] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 136.166979] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 140.616466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 140.758368] Lustre: lustre-OST0000: new disk, initializing [ 140.761541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 140.804112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 140.976553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 140.983797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 141.014193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 143.215318] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 150.175930] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 150.248191] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up) [ 150.254555] Lustre: lustre-OST0001: new disk, initializing [ 150.262159] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 150.288587] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 153.510529] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 155.661594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 155.665889] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 155.697384] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000400 [ 161.356939] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 166.849115] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 175.134435] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing check_logdir /tmp/testlogs/ [ 177.180934] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing yml_node [ 179.463173] Lustre: DEBUG MARKER: Client: 2.17.50.44 [ 180.981149] Lustre: DEBUG MARKER: MDS: 2.17.50.44 [ 182.387905] Lustre: DEBUG MARKER: OSS: 2.17.50.44 [ 183.234117] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Feb 7 01:48:15 EST 2026 [ 191.534206] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 192.108556] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 208.352356] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 208.355977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 208.362084] Lustre: Skipped 3 previous similar messages [ 208.369314] Lustre: Skipped 3 previous similar messages [ 210.317609] LustreError: 12160:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup [ 210.392500] Lustre: server umount lustre-MDT0000 complete [ 212.406653] LustreError: 6465:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770446926 with bad export cookie 13075139307429191379 [ 212.407266] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 212.412454] LustreError: 6465:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 213.481829] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 213.495531] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 213.498277] Lustre: Skipped 1 previous similar message [ 217.504737] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 217.510339] Lustre: Skipped 1 previous similar message [ 218.574935] LustreError: 12362:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 218.577593] LustreError: 12362:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 218.640493] Lustre: server umount lustre-MDT0001 complete [ 220.615178] LustreError: 12564:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 220.617866] LustreError: 12564:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 220.662134] Lustre: server umount lustre-OST0000 complete [ 222.743943] LustreError: 12764:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 222.746339] LustreError: 12764:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 222.807444] Lustre: server umount lustre-OST0001 complete [ 226.030273] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 230.386423] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 248.714142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 248.913314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 248.948259] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 248.996977] Lustre: lustre-MDT0000: new disk, initializing [ 249.043915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 249.053086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 251.310935] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 257.219052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 257.278868] Lustre: 14933:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 257.324745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 257.330259] Lustre: Skipped 1 previous similar message [ 257.377661] Lustre: lustre-MDT0001: new disk, initializing [ 257.434593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 257.454349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 257.462578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 260.026176] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 264.442385] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 266.255326] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 269.790968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 271.312913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 271.390751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 272.512285] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 276.552649] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 278.683120] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 278.778448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 280.836692] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 280.988607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 286.691096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 286.696948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.701726] Lustre: Skipped 1 previous similar message [ 286.704316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 286.709030] Lustre: Skipped 1 previous similar message [ 288.110146] LustreError: 17857:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 288.117809] LustreError: 17857:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 288.170785] Lustre: server umount lustre-OST0000 complete [ 291.808916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.814717] Lustre: Skipped 2 previous similar messages [ 296.037815] Lustre: server umount lustre-MDT0000 complete [ 296.927936] LustreError: 14943:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 296.936327] LustreError: 14943:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 6 previous similar messages [ 297.730458] LustreError: 14922:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447011 with bad export cookie 13075139307429192842 [ 297.732756] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 297.738469] LustreError: 14922:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 297.813121] LustreError: 18261:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 297.816072] LustreError: 18261:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 297.890611] Lustre: server umount lustre-MDT0001 complete [ 299.803593] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:50:12 (1770447012) [ 301.019052] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 301.827578] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:50:14 (1770447014) [ 305.006573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 305.246756] LustreError: 18929:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 305.296741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 305.299509] Lustre: Skipped 1 previous similar message [ 307.569696] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 310.755162] LustreError: 18929:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 311.743102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 313.757497] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 316.389980] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 317.983369] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 321.283829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 324.458749] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 326.855857] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 341.987700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 341.992768] LustreError: Skipped 1 previous similar message [ 341.993465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.005045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 342.008428] Lustre: Skipped 3 previous similar messages [ 345.207880] LustreError: 21192:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 345.214722] LustreError: 21192:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 345.289137] Lustre: server umount lustre-OST0000 complete [ 347.616175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 347.622779] LustreError: Skipped 1 previous similar message [ 352.225154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.235809] Lustre: Skipped 2 previous similar messages [ 353.670682] Lustre: server umount lustre-MDT0000 complete [ 355.680166] LustreError: 18909:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447069 with bad export cookie 13075139307429193724 [ 355.683039] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 355.690262] LustreError: 18909:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 355.874621] Lustre: server umount lustre-MDT0001 complete [ 361.301191] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 362.748326] Key type lgssc unregistered [ 362.936745] LNet: 22128:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.940917] LNetError: 22128:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 362.958953] LNet: Removed LNI 192.168.203.116@tcp [ 363.515179] Key type .llcrypt unregistered [ 363.518650] Key type ._llcrypt unregistered [ 375.539894] Key type ._llcrypt registered [ 375.542112] Key type .llcrypt registered [ 375.598274] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 376.094182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 376.150746] alg: No test for adler32 (adler32-zlib) [ 377.085478] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 377.214216] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 378.839370] Key type lgssc registered [ 379.358079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 383.414674] loop: module loaded [ 383.458397] loop0: detected capacity change from 0 to 204800000 [ 383.581736] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 383.588978] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 383.757492] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 387.389244] loop0: detected capacity change from 0 to 204800000 [ 387.514352] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 387.524791] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 387.535645] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 390.331882] loop0: detected capacity change from 0 to 204800000 [ 393.279682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 393.295485] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.428504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 394.469286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 394.511588] Lustre: lustre-MDT0000: new disk, initializing [ 394.566595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.578927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 396.560642] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 401.706211] loop1: detected capacity change from 0 to 204800000 [ 404.667592] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 404.800615] Lustre: lustre-OST07c6: new disk, initializing [ 404.803293] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 404.807375] Lustre: Skipped 1 previous similar message [ 404.843526] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 407.497216] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 413.713770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 413.718430] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 413.733926] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 416.621761] Lustre: 24257:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 416.626513] Lustre: 24257:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 416.629454] Lustre: 24257:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 416.632491] Lustre: 24257:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 416.636479] Lustre: 24257:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 416.639736] Lustre: 24257:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 432.939895] Lustre: 24256:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 432.945148] Lustre: 24256:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 432.947820] Lustre: 24256:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 432.950723] Lustre: 24256:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 432.953693] Lustre: 24256:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 432.956476] Lustre: 24256:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 436.668547] Lustre: 24257:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 436.671792] Lustre: 24257:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 436.674379] Lustre: 24257:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 436.677585] Lustre: 24257:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 436.682042] Lustre: 24257:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 436.686349] Lustre: 24257:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 436.689533] Lustre: 24257:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 436.692638] Lustre: 24257:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 436.696121] Lustre: 24257:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 436.698946] Lustre: 24257:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 436.701718] Lustre: 24257:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 436.704487] Lustre: 24257:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 439.772970] Lustre: 24258:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 439.777537] Lustre: 24258:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 439.780804] Lustre: 24258:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 439.785361] Lustre: 24258:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 439.790167] Lustre: 24258:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 439.796724] Lustre: 24258:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 439.799309] Lustre: 24258:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 439.803174] Lustre: 24258:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 439.806757] Lustre: 24258:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 439.809707] Lustre: 24258:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 439.812721] Lustre: 24258:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 439.815770] Lustre: 24258:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 446.571494] Lustre: 25745:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 446.575691] Lustre: 25745:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 3 previous similar messages [ 446.579364] Lustre: 25745:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 446.582941] Lustre: 25745:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 446.585466] Lustre: 25745:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 446.588324] Lustre: 25745:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 446.591521] Lustre: 25745:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 446.596089] Lustre: 25745:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 446.599369] Lustre: 25745:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 446.602769] Lustre: 25745:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 446.606462] Lustre: 25745:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 446.609147] Lustre: 25745:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 456.404584] Lustre: 24257:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 456.412068] Lustre: 24257:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 5 previous similar messages [ 456.414820] Lustre: 24257:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 456.417935] Lustre: 24257:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 456.421397] Lustre: 24257:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 456.424460] Lustre: 24257:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 456.427355] Lustre: 24257:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 456.430296] Lustre: 24257:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 456.433348] Lustre: 24257:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 456.436242] Lustre: 24257:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 456.439504] Lustre: 24257:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 456.442996] Lustre: 24257:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 500.284943] LustreError: 26440:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 500.364775] Lustre: server umount lustre-OST07c6 complete [ 520.193282] LustreError: 26941:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 520.204606] LustreError: 26941:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 520.512106] Lustre: server umount lustre-MDT0000 complete [ 536.056071] hrtimer: interrupt took 2063364 ns [ 536.981852] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 539.440536] Key type lgssc unregistered [ 539.701745] LNet: 27775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 539.712115] LNetError: 27775:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 539.728199] LNet: Removed LNI 192.168.203.116@tcp [ 540.290949] Key type .llcrypt unregistered [ 540.293713] Key type ._llcrypt unregistered [ 552.640338] Key type ._llcrypt registered [ 552.641982] Key type .llcrypt registered [ 552.711513] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 553.407962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 553.417160] alg: No test for adler32 (adler32-zlib) [ 554.365533] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 554.527324] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 556.127154] Key type lgssc registered [ 556.911973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 561.998146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 562.008583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.326592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 566.004242] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 570.916344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 571.169047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 573.454338] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 576.974650] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 579.061330] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 582.925260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 583.135813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 586.483723] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 588.265041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 589.959206] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 603.617761] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 603.626786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.641191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 606.691308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 606.691937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 606.707735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 609.468773] LustreError: 31445:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 609.569292] Lustre: server umount lustre-OST0000 complete [ 612.321259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 612.325646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 612.331682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 612.334631] Lustre: Skipped 1 previous similar message [ 616.932798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.944126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 616.956013] Lustre: Skipped 1 previous similar message [ 618.347687] LustreError: 31647:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 618.352676] LustreError: 31647:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 618.428602] Lustre: server umount lustre-MDT0000 complete [ 620.968240] LustreError: 28987:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447334 with bad export cookie 10006359798136852334 [ 620.973534] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 620.975430] LustreError: 28987:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 621.066099] LustreError: 31850:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 621.069468] LustreError: 31850:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 621.217586] Lustre: server umount lustre-MDT0001 complete [ 628.216875] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 630.163622] Key type lgssc unregistered [ 630.376944] LNet: 32381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.382987] LNetError: 32381:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.396651] LNet: Removed LNI 192.168.203.116@tcp [ 630.982513] Key type .llcrypt unregistered [ 630.987932] Key type ._llcrypt unregistered [ 643.359945] Key type ._llcrypt registered [ 643.362176] Key type .llcrypt registered [ 643.431180] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 644.105237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 644.136491] alg: No test for adler32 (adler32-zlib) [ 645.074332] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 645.235879] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 646.855160] Key type lgssc registered [ 647.635856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 652.548767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 652.573963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.906222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 656.302314] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 661.223945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 661.411233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 663.673116] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 667.159782] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 669.695378] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 674.328185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 674.617462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 678.368136] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 678.704995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 682.136359] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 694.242021] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 694.255086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.263115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 694.457474] LustreError: 35871:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 694.657584] Lustre: server umount lustre-OST0000 complete [ 697.312513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 697.323627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 697.325254] Lustre: Skipped 1 previous similar message [ 697.325894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 702.433035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 702.446071] Lustre: Skipped 2 previous similar messages [ 703.465654] LustreError: 36072:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 703.467958] LustreError: 36072:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 703.636734] Lustre: server umount lustre-MDT0000 complete [ 706.241582] LustreError: 33607:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447420 with bad export cookie 12528565120974769617 [ 706.245884] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 706.253724] LustreError: 33607:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 716.792463] LustreError: 36273:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 716.800971] LustreError: 36273:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 716.922640] Lustre: server umount lustre-MDT0001 complete [ 723.981503] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 725.767205] Key type lgssc unregistered [ 725.939873] LNet: 36806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.943857] LNetError: 36806:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.014541] LNet: Removed LNI 192.168.203.116@tcp [ 727.546561] Key type .llcrypt unregistered [ 727.548633] Key type ._llcrypt unregistered [ 742.312704] Key type ._llcrypt registered [ 742.315515] Key type .llcrypt registered [ 742.400234] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 743.123472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 743.137626] alg: No test for adler32 (adler32-zlib) [ 744.109715] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 744.286209] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 745.895196] Key type lgssc registered [ 746.522212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 751.418677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 751.438073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 752.794489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 755.423355] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 760.459399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 760.649082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 762.862889] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 766.149825] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 768.228148] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 772.040463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 772.223170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 775.559709] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 777.340718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 778.780848] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 791.010236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 791.014345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 791.024076] Lustre: Skipped 1 previous similar message [ 793.670845] LustreError: 40420:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 793.709971] Lustre: server umount lustre-MDT0000 complete [ 796.028997] LustreError: 39795:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447510 with bad export cookie 3498144035173252284 [ 796.037582] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 796.045315] LustreError: 39795:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 796.128879] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.129200] LustreError: 39792:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 796.135332] Lustre: Skipped 1 previous similar message [ 796.154741] LustreError: 39792:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 796.166871] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 796.170460] Lustre: Skipped 3 previous similar messages [ 801.251189] LustreError: 38176:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 801.251233] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 801.264080] LustreError: 38176:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 801.273732] Lustre: Skipped 1 previous similar message [ 802.184434] LustreError: 40621:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 802.187066] LustreError: 40621:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 802.236378] Lustre: server umount lustre-MDT0001 complete [ 821.046455] LustreError: 40843:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 821.049789] LustreError: 40843:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 821.131474] Lustre: server umount lustre-OST0000 complete [ 828.940948] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 830.226980] Key type lgssc unregistered [ 830.388653] LNet: 41680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.392906] LNetError: 41680:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.404614] LNet: Removed LNI 192.168.203.116@tcp [ 830.853322] Key type .llcrypt unregistered [ 830.855379] Key type ._llcrypt unregistered [ 842.128832] Key type ._llcrypt registered [ 842.131148] Key type .llcrypt registered [ 842.201794] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 842.562613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 842.630767] alg: No test for adler32 (adler32-zlib) [ 843.522832] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 843.658727] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 845.271159] Key type lgssc registered [ 845.895770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 849.719701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 849.728914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 850.999491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 853.060350] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 857.735717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 857.948376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 860.091615] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 863.166543] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 865.026640] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 868.518676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 868.719479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 872.150716] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 875.027653] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 875.884070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 886.240926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 886.250509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 886.260394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 887.265438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 887.265777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.267253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 889.650606] LustreError: 45292:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 889.701804] Lustre: server umount lustre-OST0000 complete [ 904.159979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 904.163705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 904.170878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 904.173398] Lustre: Skipped 2 previous similar messages [ 907.232509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 907.233163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 909.384504] LustreError: 45670:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 909.387235] LustreError: 45670:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 909.423240] Lustre: server umount lustre-MDT0000 complete [ 911.104808] LustreError: 43770:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447625 with bad export cookie 3270271899218525228 [ 911.108107] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 911.110758] LustreError: 43770:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 917.250867] LustreError: 45872:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 917.253760] LustreError: 45872:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 917.312481] Lustre: server umount lustre-MDT0001 complete [ 921.935750] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 923.608628] Key type lgssc unregistered [ 923.786623] LNet: 46404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.793715] LNetError: 46404:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.839079] LNet: Removed LNI 192.168.203.116@tcp [ 925.266123] Key type .llcrypt unregistered [ 925.267295] Key type ._llcrypt unregistered [ 936.520244] Key type ._llcrypt registered [ 936.521612] Key type .llcrypt registered [ 936.564092] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 936.980107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 937.032771] alg: No test for adler32 (adler32-zlib) [ 937.928268] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 938.040294] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 939.640081] Key type lgssc registered [ 940.132040] Lustre: Echo OBD driver; http://www.lustre.org/ [ 943.688376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 943.699637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 944.909071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.779216] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 951.030458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 951.196612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 952.893038] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 955.172473] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 956.845076] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 960.109094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 960.269606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 963.008816] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 965.642632] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 969.518942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 973.419353] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 975.195397] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 986.169690] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-02-07 2h01m38s [ 986.870128] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 990.177211] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 990.178446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 990.181852] Lustre: Skipped 2 previous similar messages [ 990.184546] Lustre: Skipped 1 previous similar message [ 991.201148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 991.205696] Lustre: Skipped 3 previous similar messages [ 992.535445] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 992.539089] Lustre: Skipped 1 previous similar message [ 993.859162] LustreError: 50353:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 993.895822] Lustre: server umount lustre-MDT0000 complete [ 995.317455] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 997.347627] LustreError: 48500:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 997.354790] LustreError: 48500:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 4 previous similar messages [ 998.114409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 998.219161] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 998.343430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 998.384450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 999.401877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 999.998663] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1001.319928] Lustre: DEBUG MARKER: Wait for df (35988) ... [ 1001.891364] Lustre: DEBUG MARKER: done [ 1002.231345] LustreError: 47757:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.16@tcp arrived at 1770447716 with bad export cookie 12631837229574337087 [ 1002.236710] LustreError: 47757:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1003.488907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1003.493056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1003.499927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1004.518237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.056487] LustreError: 51397:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1005.060599] LustreError: 51397:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1005.106459] Lustre: server umount lustre-OST0000 complete [ 1009.636198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1009.636339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1009.642566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1009.647320] Lustre: Skipped 3 previous similar messages [ 1012.732649] LustreError: 51600:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1012.735217] LustreError: 51600:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1012.762638] Lustre: server umount lustre-MDT0000 complete [ 1014.170023] LustreError: 47758:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447728 with bad export cookie 12631837229574337528 [ 1014.173164] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1014.174832] LustreError: 47758:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1014.289029] Lustre: server umount lustre-MDT0001 complete [ 1018.347803] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1019.379956] Key type lgssc unregistered [ 1019.502383] LNet: 52333:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.504956] LNetError: 52333:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.517363] LNet: Removed LNI 192.168.203.116@tcp [ 1019.808767] Key type .llcrypt unregistered [ 1019.809878] Key type ._llcrypt unregistered [ 1035.856081] Key type ._llcrypt registered [ 1035.857306] Key type .llcrypt registered [ 1035.902924] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1036.303423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1036.362883] alg: No test for adler32 (adler32-zlib) [ 1037.230457] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 1037.335500] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1038.943666] Key type lgssc registered [ 1039.460511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1042.725109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1042.732926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1043.879687] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1043.892039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1043.968558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.494971] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1049.135911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1049.179191] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1049.192825] Lustre: 54498:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1049.301905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1050.676289] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1052.694958] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1054.132393] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1056.938192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1057.031518] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1057.125849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1059.311196] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1060.709047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1061.527280] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1065.509677] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-02-07 2h02m58s [ 1066.247645] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1069.366640] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1070.472656] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1070.508471] LustreError: 54505:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1090.867736] Lustre: lustre-MDT0000: Client 5df3e9ce-8a07-4026-8609-1fc34fbcb1cf (at 192.168.203.16@tcp) reconnecting [ 1110.563375] LustreError: 54505:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1111.665134] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1112.946713] Lustre: DEBUG MARKER: done [ 1116.134577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1116.138247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.143740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1117.158088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1117.159375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.159890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1121.015074] LustreError: 57528:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1121.105289] Lustre: server umount lustre-OST0000 complete [ 1126.370772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1126.377351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.385077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1126.388014] Lustre: Skipped 1 previous similar message [ 1129.023433] LustreError: 57730:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1129.025766] LustreError: 57730:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1129.051800] Lustre: server umount lustre-MDT0000 complete [ 1130.696708] LustreError: 54491:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447844 with bad export cookie 16464294269411389301 [ 1130.700161] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1130.704386] LustreError: 54491:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1130.791602] LustreError: 57932:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1130.794055] LustreError: 57932:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1130.887318] Lustre: server umount lustre-MDT0001 complete [ 1135.741926] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1137.316554] Key type lgssc unregistered [ 1137.477047] LNet: 58462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1137.481073] LNetError: 58462:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1137.492879] LNet: Removed LNI 192.168.203.116@tcp [ 1137.879167] Key type .llcrypt unregistered [ 1137.880869] Key type ._llcrypt unregistered [ 1158.001518] Key type ._llcrypt registered [ 1158.002934] Key type .llcrypt registered [ 1158.049621] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1158.498318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1158.506781] alg: No test for adler32 (adler32-zlib) [ 1159.383740] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 1159.477507] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1161.063164] Key type lgssc registered [ 1161.525191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1162.785610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1162.794596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1163.919115] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1163.938097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1164.038279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1164.532623] Lustre: Failing over lustre-MDT0000 [ 1164.605267] LustreError: 60662:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1164.677628] Lustre: server umount lustre-MDT0000 complete [ 1167.496231] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:04:40 (1770447880) [ 1170.435614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1170.620508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1172.226247] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1175.936451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1175.975813] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1175.989233] Lustre: 61216:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1176.097851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1177.531090] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1179.458524] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1180.854968] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1183.344279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1183.417954] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1183.489448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1185.661194] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1187.875071] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1188.895937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1196.261704] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1199.075188] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1199.076252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1199.084820] Lustre: Skipped 2 previous similar messages [ 1199.092198] Lustre: Skipped 2 previous similar messages [ 1204.193832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1204.196498] Lustre: Skipped 2 previous similar messages [ 1204.544253] LustreError: 63584:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1204.546793] LustreError: 63584:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1204.575621] Lustre: server umount lustre-MDT0000 complete [ 1205.955227] LustreError: 61208:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447919 with bad export cookie 926507340839338475 [ 1205.958733] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1205.959689] LustreError: 61208:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1206.030347] LustreError: 63786:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1206.032987] LustreError: 63786:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1206.084599] Lustre: server umount lustre-MDT0001 complete [ 1207.312733] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1209.119452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1212.428490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1215.629733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1215.675431] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1217.131785] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1220.868806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1221.028529] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1222.693186] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1225.003260] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1225.120145] Lustre: 60145:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770447923/real 1770447923] req@0000000055911772 x1856449148910336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1770447939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1226.143189] Lustre: 60144:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770447924/real 1770447924] req@00000000e2172e8b x1856449148910464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1770447940 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1226.210816] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1226.835492] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1229.285808] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1230.303316] Lustre: 60142:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770447928/real 1770447928] req@000000000d929dae x1856449148910592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1770447944 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1231.328431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1231.332744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.338448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1234.406355] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1234.406545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1234.413964] Lustre: Skipped 2 previous similar messages [ 1235.711880] LustreError: 66039:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1235.714513] LustreError: 66039:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1235.752912] Lustre: server umount lustre-MDT0000 complete [ 1237.170389] LustreError: 61936:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447951 with bad export cookie 926507340839340750 [ 1237.173591] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1237.176609] LustreError: 61936:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1237.292698] Lustre: server umount lustre-MDT0001 complete [ 1239.852450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1243.589921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1247.647719] LustreError: 60141:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002bb1ae33 x1856449148930048/t0(0) o250->MGC192.168.203.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1247.781372] LustreError: 62838:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.789722] LustreError: 62838:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 1247.808935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1247.811334] Lustre: Skipped 1 previous similar message [ 1247.831732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1249.334087] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1252.834886] LustreError: 62838:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1252.836510] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.848251] Lustre: Skipped 1 previous similar message [ 1252.865754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1254.053866] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1254.501946] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1255.074815] Lustre: 60145:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770447953/real 1770447953] req@000000003ac10c51 x1856449148929664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1770447969 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1255.088668] Lustre: 60145:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1256.563077] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1258.161684] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1261.279145] Lustre: 60143:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770447958/real 1770447958] req@000000007e2e1f04 x1856449148929920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1770447974 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1261.290207] Lustre: 60143:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1263.584167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1263.587423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.593323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1266.367019] LustreError: 68490:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1266.369814] LustreError: 68490:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1266.398480] Lustre: server umount lustre-MDT0000 complete [ 1267.759453] LustreError: 61209:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447981 with bad export cookie 926507340839342122 [ 1267.763320] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1267.767030] LustreError: 61209:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1267.892689] Lustre: server umount lustre-MDT0001 complete [ 1270.212782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1271.343352] Lustre: DEBUG MARKER: files compared the same [ 1281.206780] LustreError: 69284:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1281.209047] LustreError: 69284:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1281.273342] Lustre: server umount lustre-OST0000 complete [ 1287.608672] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1288.758020] Key type lgssc unregistered [ 1288.890421] LNet: 70123:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.893039] LNetError: 70123:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1288.901446] LNet: Removed LNI 192.168.203.116@tcp [ 1289.213223] Key type .llcrypt unregistered [ 1289.214718] Key type ._llcrypt unregistered [ 1298.829139] Key type ._llcrypt registered [ 1298.830475] Key type .llcrypt registered [ 1298.869818] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1299.189435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1299.206060] alg: No test for adler32 (adler32-zlib) [ 1300.080549] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 1300.176626] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1301.767262] Key type lgssc registered [ 1302.201063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1305.423103] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1305.433685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1306.620028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1308.147920] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug +malloc all [ 1311.462049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1311.576366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1313.137401] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug +malloc all [ 1315.139421] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1316.482912] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1319.151916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1319.256647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1321.662922] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug +malloc all [ 1323.801985] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1331.555181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1341.920531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.920557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1341.926066] Lustre: Skipped 1 previous similar message [ 1341.928637] LustreError: Skipped 1 previous similar message [ 1341.930542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1341.934562] Lustre: Skipped 1 previous similar message [ 1343.273793] LustreError: 73755:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1343.305229] Lustre: server umount lustre-OST0000 complete [ 1347.039884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.040166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1347.045033] Lustre: Skipped 1 previous similar message [ 1347.048776] Lustre: Skipped 1 previous similar message [ 1350.719570] LustreError: 73958:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1350.721931] LustreError: 73958:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1350.750814] Lustre: server umount lustre-MDT0000 complete [ 1352.159594] LustreError: 71492:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.166687] LustreError: 71492:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 1352.242587] LustreError: 73110:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770448066 with bad export cookie 4772839758282449247 [ 1352.244229] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1352.249871] LustreError: 73110:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1352.323132] LustreError: 74160:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1352.324929] LustreError: 74160:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1352.389948] Lustre: server umount lustre-MDT0001 complete [ 1356.435452] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1357.551204] Key type lgssc unregistered [ 1357.695621] LNet: 74691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.699337] LNetError: 74691:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.707527] LNet: Removed LNI 192.168.203.116@tcp [ 1358.034695] Key type .llcrypt unregistered [ 1358.036219] Key type ._llcrypt unregistered [ 1367.253659] Key type ._llcrypt registered [ 1367.254747] Key type .llcrypt registered [ 1367.294316] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1367.623314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1367.662504] alg: No test for adler32 (adler32-zlib) [ 1368.538875] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 1368.622283] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1370.207141] Key type lgssc registered [ 1370.566575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1373.326298] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1373.334235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1388.767775] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1393.887187] LustreError: 76039:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 1420.511177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1422.550438] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1424.426870] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1427.059833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1427.115495] Lustre: *** cfs_fail_loc=706, val=0*** [ 1427.248708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.279262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1428.604919] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1431.565779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1431.714245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1433.104325] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1435.046808] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1436.362238] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1437.152427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1437.156076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1437.162868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1437.663830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1437.666988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1437.671912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1441.249260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1441.252400] Lustre: Skipped 1 previous similar message [ 1443.180275] LustreError: 78356:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1443.216390] Lustre: server umount lustre-OST0000 complete [ 1446.369045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.370206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1446.373471] Lustre: Skipped 1 previous similar message [ 1450.617223] LustreError: 78559:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1450.619261] LustreError: 78559:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1450.641210] Lustre: server umount lustre-MDT0000 complete [ 1451.487957] LustreError: 77012:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1451.492441] LustreError: 77012:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 1451.821799] LustreError: 76047:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770448165 with bad export cookie 12776931126689982605 [ 1451.823500] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1451.826510] LustreError: 76047:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1451.887142] LustreError: 78763:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1451.889637] LustreError: 78763:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1451.932577] Lustre: server umount lustre-MDT0001 complete [ 1455.148929] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1456.083803] Key type lgssc unregistered [ 1456.205345] LNet: 79295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1456.208503] LNetError: 79295:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1456.222303] LNet: Removed LNI 192.168.203.116@tcp [ 1456.508507] Key type .llcrypt unregistered [ 1456.509434] Key type ._llcrypt unregistered [ 1465.130956] Key type ._llcrypt registered [ 1465.131917] Key type .llcrypt registered [ 1465.165513] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1465.455630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1465.496197] alg: No test for adler32 (adler32-zlib) [ 1466.343073] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 1466.423477] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1468.007072] Key type lgssc registered [ 1468.343990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1470.784116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1470.789457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.021382] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1475.409258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1475.556658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1476.759391] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1479.566357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1479.680312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1481.578326] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1484.455601] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1484.461302] Lustre: MGS: Client c9a19015-0fd5-4deb-b797-627fcebd7e89 (at 0@lo) reconnecting [ 1484.544169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1484.574853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1485.922859] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1495.008462] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1495.010991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1495.015526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1496.034060] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1496.036159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1496.036275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1496.043059] Lustre: Skipped 1 previous similar message [ 1499.892144] LustreError: 83303:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1499.938036] Lustre: server umount lustre-OST0000 complete [ 1506.272707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1506.272783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1506.281055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1506.283175] Lustre: Skipped 1 previous similar message [ 1507.330262] LustreError: 83505:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1507.332585] LustreError: 83505:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1507.342636] Lustre: server umount lustre-MDT0000 complete [ 1508.919328] LustreError: 83708:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1508.921206] LustreError: 83708:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1508.964650] Lustre: server umount lustre-MDT0001 complete [ 1510.283331] Lustre: server umount MGS complete [ 1514.225142] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:10:27 (1770448227) [ 1523.764726] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 1526.760488] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1539.121836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1540.420672] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1544.303354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1544.388567] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1544.408166] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1544.444334] Lustre: lustre-MDT0001: new disk, initializing [ 1544.469653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1544.478795] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1544.481694] Lustre: 88403:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1544.484656] LustreError: 88403:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1545.784384] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1550.057633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1550.151154] Lustre: lustre-OST0000: new disk, initializing [ 1550.153402] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1550.173409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1551.961613] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1554.968132] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1556.448316] Lustre: 89800:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000e46bebb4 x1856449470825728/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1770448286 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1557.233408] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1557.239118] Lustre: MGS: Client dc6d5176-ab1d-4d82-b45c-52da5b9c7da4 (at 0@lo) reconnecting [ 1557.253717] Lustre: 87691:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1557.262636] LustreError: 87691:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1557.274542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1557.311330] Lustre: lustre-MDT0000: new disk, initializing [ 1557.366873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1558.431808] Lustre: 90099:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000b5b99abb x1856449470830848/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1770448288 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1558.440286] Lustre: 90099:0:(client.c:1637:after_reply()) Skipped 1 previous similar message [ 1558.495554] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1558.678672] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1565.538045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1565.541838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1567.521121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1567.523321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1567.524797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1588.192816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1588.196332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1588.200952] Lustre: Skipped 1 previous similar message [ 1588.202987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1594.850551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1594.852621] Lustre: Skipped 3 previous similar messages [ 1602.015833] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1602.056128] LustreError: 90630:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1602.059698] LustreError: 90630:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1602.114250] Lustre: server umount lustre-OST0000 complete [ 1605.088709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1605.089081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1605.092775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1605.097340] Lustre: Skipped 2 previous similar messages [ 1605.099847] Lustre: Skipped 2 previous similar messages [ 1609.860696] LustreError: 90834:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1609.863639] LustreError: 90834:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1609.872608] Lustre: server umount lustre-MDT0000 complete [ 1610.209415] LustreError: 88374:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1610.217481] LustreError: 88374:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 6 previous similar messages [ 1617.582185] Lustre: server umount lustre-MDT0001 complete [ 1618.897996] LustreError: 91238:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1618.901198] LustreError: 91238:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1618.926894] Lustre: server umount MGS complete [ 1622.526335] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:12:15 (1770448335) [ 1628.063584] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1629.100722] Key type lgssc unregistered [ 1629.232354] LNet: 92572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.234912] LNetError: 92572:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1629.245300] LNet: Removed LNI 192.168.203.116@tcp [ 1629.569814] Key type .llcrypt unregistered [ 1629.571208] Key type ._llcrypt unregistered [ 1637.118695] Key type ._llcrypt registered [ 1637.120156] Key type .llcrypt registered [ 1637.162381] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1637.574491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1637.581278] alg: No test for adler32 (adler32-zlib) [ 1638.438605] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 1638.521688] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1640.103225] Key type lgssc registered [ 1640.496902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1643.138777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1643.146479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1644.316081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.666390] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1648.650162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1648.774705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1650.205337] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1652.065516] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1653.226086] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1655.455838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1655.555368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.327162] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1659.056111] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1670.950191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1679.327925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1679.330632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1679.334670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1681.377113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.377788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1681.378797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1683.181074] LustreError: 96016:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1683.222195] Lustre: server umount lustre-OST0000 complete [ 1684.959741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1684.963188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1684.969751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1684.971619] Lustre: Skipped 1 previous similar message [ 1690.620273] LustreError: 96217:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1690.622572] LustreError: 96217:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1690.646794] Lustre: server umount lustre-MDT0000 complete [ 1691.616067] LustreError: 93751:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1691.624339] LustreError: 93751:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 1691.876118] LustreError: 93732:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770448405 with bad export cookie 6648977997442557509 [ 1691.878811] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1691.879790] LustreError: 93732:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1691.949883] LustreError: 96419:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1691.952221] LustreError: 96419:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1692.011778] Lustre: server umount lustre-MDT0001 complete [ 1695.742948] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1696.972120] Key type lgssc unregistered [ 1697.149369] LNet: 96950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1697.154282] LNetError: 96950:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1697.175893] LNet: Removed LNI 192.168.203.116@tcp [ 1697.548487] Key type .llcrypt unregistered [ 1697.549897] Key type ._llcrypt unregistered [ 1704.504387] Key type ._llcrypt registered [ 1704.505844] Key type .llcrypt registered [ 1704.556488] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1704.889231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1704.954811] alg: No test for adler32 (adler32-zlib) [ 1705.850202] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 1705.947621] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1707.527151] Key type lgssc registered [ 1707.927608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1710.186811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1710.186819] LustreError: 97989:0:(tgt_mount.c:2439:server_fill_super()) cfs_race id 716 sleeping [ 1710.189616] Lustre: Skipped 1 previous similar message [ 1710.189633] LustreError: 97990:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 waking [ 1710.195942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1710.197223] LustreError: 97989:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 1710.203287] LustreError: 97989:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1710.208805] LustreError: 97989:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1711.382920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1714.355256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1714.484462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1715.955697] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1717.710845] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing lsmod [ 1718.943729] LustreError: 99163:0:(tgt_mount.c:2439:server_fill_super()) cfs_race id 716 sleeping [ 1718.948578] LustreError: 99164:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 waking [ 1718.953571] LustreError: 99163:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 1718.956049] LustreError: 99163:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1718.960787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1718.961896] LustreError: 99163:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1719.130241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1720.389061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1720.391991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1720.400076] Lustre: Skipped 1 previous similar message [ 1724.513076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1724.514912] Lustre: Skipped 4 previous similar messages [ 1726.523620] LustreError: 99351:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1726.550101] Lustre: server umount lustre-MDT0000 complete [ 1727.773127] LustreError: 98004:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770448441 with bad export cookie 2780925173354167646 [ 1727.777416] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1727.777720] LustreError: 98004:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1728.607202] LustreError: 98460:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003846baa0 x1856449721416960/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1728.616957] LustreError: 99554:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1728.619750] LustreError: 99554:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1728.659178] Lustre: server umount lustre-MDT0001 complete [ 1739.250719] LustreError: 99755:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1739.252798] LustreError: 99755:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1739.294752] Lustre: server umount lustre-OST0000 complete [ 1742.117017] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1742.271935] LustreError: 100231:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1742.279567] LustreError: 100231:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 1742.296101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1743.568939] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1746.339026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1746.452520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1747.710489] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1749.455154] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1750.617696] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1752.768218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1754.658613] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1756.390088] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1767.205964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1777.634157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1777.634649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1777.638364] Lustre: Skipped 1 previous similar message [ 1777.643893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1777.645924] Lustre: Skipped 1 previous similar message [ 1782.752462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1782.757153] Lustre: Skipped 1 previous similar message [ 1787.359193] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1787.381109] LustreError: 102494:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1787.383829] LustreError: 102494:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1787.422508] Lustre: server umount lustre-OST0000 complete [ 1792.480035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1792.483636] LustreError: Skipped 1 previous similar message [ 1792.485756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1792.491613] Lustre: Skipped 1 previous similar message [ 1792.494289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1803.231121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1803.232592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1803.236296] Lustre: Skipped 4 previous similar messages [ 1803.265996] LustreError: 102697:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1803.267734] LustreError: 102697:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1803.296000] Lustre: server umount lustre-MDT0000 complete [ 1804.600320] LustreError: 102395:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770448518 with bad export cookie 2780925173354168318 [ 1804.603441] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1804.604487] LustreError: 102395:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1804.719637] Lustre: server umount lustre-MDT0001 complete [ 1808.040333] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1809.057409] Key type lgssc unregistered [ 1809.188386] LNet: 103430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1809.191654] LNetError: 103430:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1809.203496] LNet: Removed LNI 192.168.203.116@tcp [ 1809.507054] Key type .llcrypt unregistered [ 1809.508562] Key type ._llcrypt unregistered [ 1818.605486] Key type ._llcrypt registered [ 1818.607017] Key type .llcrypt registered [ 1818.648138] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1819.062662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1819.076351] alg: No test for adler32 (adler32-zlib) [ 1819.931859] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 1820.021329] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1821.607158] Key type lgssc registered [ 1821.957525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1824.508479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1824.515859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1825.669377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1827.061526] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1830.557440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1830.730684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1832.136128] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1834.184174] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1835.559420] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1838.167510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1838.310916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1840.173509] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1842.153549] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1850.601587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1854.996304] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1856.130076] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1861.089111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1861.091576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.093643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1861.109667] Lustre: Skipped 1 previous similar message [ 1864.114959] LustreError: 107165:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1864.195920] Lustre: server umount lustre-OST0000 complete [ 1871.330589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1871.339880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1871.342663] Lustre: Skipped 1 previous similar message [ 1876.449762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1876.460253] Lustre: Skipped 1 previous similar message [ 1881.056305] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1881.099952] LustreError: 107370:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1881.102104] LustreError: 107370:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1881.134862] Lustre: server umount lustre-MDT0000 complete [ 1881.568410] LustreError: 104802:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1881.576082] LustreError: 104802:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 1882.640264] LustreError: 107019:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770448596 with bad export cookie 3190785386539592289 [ 1882.643616] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1882.643910] LustreError: 107019:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1882.711953] LustreError: 107572:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1882.714735] LustreError: 107572:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1882.767954] Lustre: server umount lustre-MDT0001 complete [ 1886.493376] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1887.611410] Key type lgssc unregistered [ 1887.735483] LNet: 108104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1887.738251] LNetError: 108104:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1887.750442] LNet: Removed LNI 192.168.203.116@tcp [ 1888.093423] Key type .llcrypt unregistered [ 1888.095056] Key type ._llcrypt unregistered [ 1895.263334] Key type ._llcrypt registered [ 1895.264493] Key type .llcrypt registered [ 1895.304059] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1895.653157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1895.725925] alg: No test for adler32 (adler32-zlib) [ 1896.614755] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 1896.716941] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1898.303196] Key type lgssc registered [ 1898.725826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1901.725411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1901.732924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1902.940659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1904.420866] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1907.500713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1907.625951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1909.093444] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1911.113782] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1912.680351] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1915.505566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1915.719600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1917.848760] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1919.766952] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1927.018926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1933.281285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1933.285175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1933.291370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1937.376844] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1937.378825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1937.379461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.309370] LustreError: 111550:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1939.345964] Lustre: server umount lustre-OST0000 complete [ 1942.496514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1942.497392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1942.501478] Lustre: Skipped 1 previous similar message [ 1942.503314] Lustre: Skipped 1 previous similar message [ 1946.750343] LustreError: 111752:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1946.752981] LustreError: 111752:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1946.781642] Lustre: server umount lustre-MDT0000 complete [ 1947.616356] LustreError: 109286:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1947.621734] LustreError: 109286:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 1948.118591] LustreError: 109267:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770448662 with bad export cookie 15173314169532021937 [ 1948.123308] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1948.123482] LustreError: 109267:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1948.194131] LustreError: 111955:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1948.197127] LustreError: 111955:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1948.251287] Lustre: server umount lustre-MDT0001 complete [ 1951.822264] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1952.829684] Key type lgssc unregistered [ 1952.954407] LNet: 112487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1952.957441] LNetError: 112487:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1952.964317] LNet: Removed LNI 192.168.203.116@tcp [ 1953.269172] Key type .llcrypt unregistered [ 1953.270423] Key type ._llcrypt unregistered [ 1962.794694] Key type ._llcrypt registered [ 1962.796653] Key type .llcrypt registered [ 1962.837712] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1963.190800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1963.204881] alg: No test for adler32 (adler32-zlib) [ 1964.084419] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 1964.198404] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1965.791125] Key type lgssc registered [ 1966.202273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1969.046963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1969.053534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1970.232088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1971.763676] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1974.881362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1975.059985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1976.556455] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1978.771250] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1980.170528] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1982.654400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1982.783179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.855778] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1986.897909] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1995.112942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 2000.063847] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2001.374352] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2001.377061] Lustre: Skipped 2 previous similar messages [ 2002.936046] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2002.943258] Lustre: Skipped 2 previous similar messages [ 2009.601834] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2009.616389] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2014.311234] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.16@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2014.315513] Lustre: Skipped 2 previous similar messages [ 2015.160791] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.16@tcp 192.168.0.[2,10]@tcp [ 2021.764878] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.16/24@tcp [ 2021.770651] Lustre: Skipped 1 previous similar message [ 2026.976637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2026.981571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.990658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2030.050668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2030.051144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2030.054316] LustreError: Skipped 1 previous similar message [ 2030.062567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2031.073163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2031.074932] Lustre: Skipped 1 previous similar message [ 2032.044586] LustreError: 117572:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2032.089944] Lustre: server umount lustre-OST0000 complete [ 2036.192812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2036.193241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2036.200747] Lustre: Skipped 1 previous similar message [ 2036.204489] Lustre: Skipped 1 previous similar message [ 2039.498212] LustreError: 117773:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2039.500195] LustreError: 117773:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2039.538052] Lustre: server umount lustre-MDT0000 complete [ 2041.068476] LustreError: 113839:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770448755 with bad export cookie 15490152366128714979 [ 2041.070709] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2041.073914] LustreError: 113839:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2041.146436] LustreError: 117975:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2041.149365] LustreError: 117975:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2041.206073] Lustre: server umount lustre-MDT0001 complete [ 2045.565269] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2046.781713] Key type lgssc unregistered [ 2046.942590] LNet: 118506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.946743] LNetError: 118506:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.956917] LNet: Removed LNI 192.168.203.116@tcp [ 2047.311819] Key type .llcrypt unregistered [ 2047.313403] Key type ._llcrypt unregistered [ 2060.303469] Key type ._llcrypt registered [ 2060.305118] Key type .llcrypt registered [ 2060.346869] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2060.662729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2060.670344] alg: No test for adler32 (adler32-zlib) [ 2061.526613] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 2061.618485] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2063.207108] Key type lgssc registered [ 2063.582309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2066.335540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2066.344982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.521273] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2067.526285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2067.529891] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.16/24@tcp [ 2069.027134] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2072.451318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2072.576785] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2072.587231] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.16/24@tcp [ 2072.612210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2074.081889] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2076.302284] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2077.929729] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2080.600964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2080.757291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2082.935605] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2084.919987] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2093.039961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2103.265924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.267764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2103.267986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2103.270447] Lustre: Skipped 1 previous similar message [ 2104.880070] LustreError: 122342:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2104.925611] Lustre: server umount lustre-OST0000 complete [ 2108.385149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.386563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2108.391691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2108.393909] LustreError: Skipped 1 previous similar message [ 2108.396049] Lustre: Skipped 1 previous similar message [ 2112.529341] LustreError: 122543:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2112.531425] LustreError: 122543:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2112.563530] Lustre: server umount lustre-MDT0000 complete [ 2113.504074] LustreError: 120025:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2113.510077] LustreError: 120025:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 2113.853526] LustreError: 120052:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770448827 with bad export cookie 6692424767245924444 [ 2113.857053] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2113.857871] LustreError: 120052:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2113.926160] LustreError: 122745:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2113.928410] LustreError: 122745:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2113.979679] Lustre: server umount lustre-MDT0001 complete [ 2117.824223] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2118.937025] Key type lgssc unregistered [ 2119.064327] LNet: 123275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2119.066934] LNetError: 123275:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2119.075295] LNet: Removed LNI 192.168.203.116@tcp [ 2119.389289] Key type .llcrypt unregistered [ 2119.390435] Key type ._llcrypt unregistered [ 2128.938841] Key type ._llcrypt registered [ 2128.939931] Key type .llcrypt registered [ 2128.980518] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 2135.853598] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2136.233694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2136.268699] alg: No test for adler32 (adler32-zlib) [ 2137.136352] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 2137.245521] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2138.847196] Key type lgssc registered [ 2139.255376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2152.569672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2152.577905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2153.692161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2153.711497] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2153.756461] Lustre: lustre-MDT0000: new disk, initializing [ 2153.796525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2153.804416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2155.403584] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2160.423310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2160.471616] Lustre: 126611:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2160.494799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2160.496610] Lustre: Skipped 1 previous similar message [ 2160.536344] Lustre: lustre-MDT0001: new disk, initializing [ 2160.578245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2160.592993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2160.597855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2162.471652] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2165.831177] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2167.211216] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2169.614617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2169.721137] Lustre: lustre-OST0000: new disk, initializing [ 2169.723820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2169.760741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2171.468629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2171.472682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2171.524834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2171.740699] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2175.013855] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2176.449275] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2176.526570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2178.061770] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2178.139142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2182.116259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2182.117312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.118662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2182.119181] LustreError: Skipped 1 previous similar message [ 2182.123450] Lustre: Skipped 1 previous similar message [ 2187.233643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2187.236015] Lustre: Skipped 2 previous similar messages [ 2192.354725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2192.356799] Lustre: Skipped 1 previous similar message [ 2193.375102] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2193.392792] LustreError: 129541:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2193.432500] Lustre: server umount lustre-OST0000 complete [ 2196.447853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2196.450757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2196.454549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2197.473727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2202.591946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2202.594196] Lustre: Skipped 3 previous similar messages [ 2209.247150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2209.292562] LustreError: 129744:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2209.294339] LustreError: 129744:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2209.329227] Lustre: server umount lustre-MDT0000 complete [ 2211.103291] LustreError: 126602:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770448925 with bad export cookie 5206477656385874017 [ 2211.105982] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2211.107661] LustreError: 126602:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2211.185857] LustreError: 129946:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2211.188658] LustreError: 129946:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2211.261591] Lustre: server umount lustre-MDT0001 complete [ 2214.595143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.764244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2216.206360] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2219.248201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2219.390793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2220.937283] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2222.996141] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2224.331677] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2226.744976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2228.834690] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2230.607771] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2231.888902] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2240.119240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2241.667379] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2241.743155] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2245.473720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2245.549228] Lustre: lustre-OST0001: new disk, initializing [ 2245.551623] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2245.591555] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2245.594474] Lustre: Skipped 1 previous similar message [ 2247.600328] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2251.051637] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2251.783067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2251.786891] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2251.824917] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2252.153523] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2253.573072] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2253.649812] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2254.956851] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b8bd23c4000.ost_server_uuid 50 [ 2255.534628] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b8bd23c4000.ost_server_uuid in FULL state after 0 sec [ 2261.992507] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2261.992778] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2261.993558] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2261.993580] Lustre: Skipped 2 previous similar messages [ 2261.995636] LustreError: Skipped 1 previous similar message [ 2262.000034] Lustre: Skipped 1 previous similar message [ 2264.047072] LustreError: 134724:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2264.049827] LustreError: 134724:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2264.077124] Lustre: server umount lustre-OST0001 complete [ 2266.079771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2266.083100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2267.107948] LustreError: 132013:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2267.113189] LustreError: 132013:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 2271.533102] LustreError: 134926:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2271.538151] LustreError: 134926:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2271.577298] Lustre: server umount lustre-OST0000 complete [ 2275.807804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2275.811186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2275.815665] Lustre: Skipped 1 previous similar message [ 2279.016624] Lustre: server umount lustre-MDT0000 complete [ 2280.391181] LustreError: 130402:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770448994 with bad export cookie 5206477656385874885 [ 2280.393422] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2280.395409] LustreError: 130402:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2280.465072] LustreError: 135330:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2280.467959] LustreError: 135330:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2280.517885] Lustre: server umount lustre-MDT0001 complete [ 2287.329624] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2288.428471] Key type lgssc unregistered [ 2288.560350] LNet: 136319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2288.563636] LNetError: 136319:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2288.570496] LNet: Removed LNI 192.168.203.116@tcp [ 2288.906185] Key type .llcrypt unregistered [ 2288.907795] Key type ._llcrypt unregistered [ 2302.413597] Key type ._llcrypt registered [ 2302.415103] Key type .llcrypt registered [ 2302.452743] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 2308.647786] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2308.974298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2309.029808] alg: No test for adler32 (adler32-zlib) [ 2309.879747] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 2309.963946] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2311.543143] Key type lgssc registered [ 2311.906666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2323.627306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2323.633839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2324.731920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2324.743261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2324.780667] Lustre: lustre-MDT0000: new disk, initializing [ 2324.806307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2324.814668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2326.148983] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2330.343278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2330.381044] Lustre: 140415:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2330.402974] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2330.404783] Lustre: Skipped 1 previous similar message [ 2330.436314] Lustre: lustre-MDT0001: new disk, initializing [ 2330.466124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2330.476162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2330.479176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2331.742450] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2334.822537] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2336.015440] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2338.131904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2338.221596] Lustre: lustre-OST0000: new disk, initializing [ 2338.223286] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2338.247206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2340.070343] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2341.235748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2341.239361] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2341.267792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2343.211728] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2345.143883] Lustre: Failing over lustre-OST0000 [ 2345.174096] LustreError: 142847:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2345.208511] Lustre: server umount lustre-OST0000 complete [ 2358.038366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2358.133019] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2358.137489] Lustre: Skipped 1 previous similar message [ 2358.143836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.150992] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2358.822969] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2359.998542] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2361.235520] Lustre: Failing over lustre-MDT0000 [ 2361.302560] LustreError: 143816:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2361.304531] LustreError: 143816:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2361.313611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2361.316169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.338259] Lustre: server umount lustre-MDT0000 complete [ 2362.271142] Lustre: 138641:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770449060/real 1770449060] req@00000000cee0750e x1856450354766336/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1770449076 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2363.361382] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2363.365433] Lustre: Skipped 1 previous similar message [ 2367.455148] Lustre: 138644:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770449065/real 1770449065] req@00000000bcc12113 x1856450354768128/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1770449081 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2368.279335] LustreError: 142162:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2368.283829] LustreError: 142162:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 6 previous similar messages [ 2372.575119] Lustre: 138644:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770449070/real 1770449070] req@0000000017acc774 x1856450354769152/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1770449086 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2373.400675] LustreError: 142162:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.407728] LustreError: 142162:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 2374.257369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.292403] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2374.296537] Lustre: 140418:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2374.360768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2374.377278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2374.399719] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2375.592983] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2378.519095] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2379.749507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2379.756266] Lustre: lustre-OST0000: Recovery over after 0:21, of 3 clients 3 recovered and 0 were evicted. [ 2379.758590] Lustre: Skipped 1 previous similar message [ 2379.779518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2384.866256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2384.866629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2384.870421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2384.871686] LustreError: Skipped 1 previous similar message [ 2384.874892] Lustre: Skipped 1 previous similar message [ 2386.795134] LustreError: 144699:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2386.796969] LustreError: 144699:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2386.825305] Lustre: server umount lustre-OST0000 complete [ 2389.985368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2389.985804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2389.986259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2389.991145] Lustre: Skipped 2 previous similar messages [ 2394.236091] LustreError: 144902:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2394.238512] LustreError: 144902:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2394.268617] Lustre: server umount lustre-MDT0000 complete [ 2395.103755] LustreError: 140425:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.109330] LustreError: 140425:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 4 previous similar messages [ 2395.483043] LustreError: 140407:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770449109 with bad export cookie 8253371273361196807 [ 2395.486560] LustreError: 140407:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2395.486708] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2395.598642] Lustre: server umount lustre-MDT0001 complete [ 2399.076095] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2400.005495] Key type lgssc unregistered [ 2400.124302] LNet: 145635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2400.127209] LNetError: 145635:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2400.137276] LNet: Removed LNI 192.168.203.116@tcp [ 2400.399918] Key type .llcrypt unregistered [ 2400.401118] Key type ._llcrypt unregistered [ 2419.807179] Key type ._llcrypt registered [ 2419.808140] Key type .llcrypt registered [ 2419.844702] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2420.269557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2420.285645] alg: No test for adler32 (adler32-zlib) [ 2421.129139] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 2421.211859] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2422.783171] Key type lgssc registered [ 2423.132996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2425.638120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2425.645026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2426.793089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2427.986340] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2430.982756] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2432.956861] LustreError: 147162:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2432.960243] LustreError: 147162:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2432.963951] LustreError: 147162:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2433.023616] LustreError: 147683:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2433.073452] Lustre: server umount lustre-MDT0000 complete [ 2437.729698] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 2440.694421] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2452.155389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.239518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2452.251340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2452.286804] Lustre: lustre-MDT0000: new disk, initializing [ 2452.309371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2452.315166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2453.594400] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2457.696547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2457.741950] Lustre: 150311:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2457.767193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2457.768716] Lustre: Skipped 1 previous similar message [ 2457.808913] Lustre: lustre-MDT0001: new disk, initializing [ 2457.843068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2457.854828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2457.857481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2459.153170] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2462.170364] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2463.288247] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2465.441218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2465.549030] Lustre: lustre-OST0000: new disk, initializing [ 2465.550691] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2465.576552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2467.265068] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2470.254443] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2471.928864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2471.932628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2471.969094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2477.475598] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2478.001386] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:26:31 (1770449191) [ 2539.999552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2540.002967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2540.006615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2543.583722] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.583788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2543.588374] Lustre: Skipped 1 previous similar message [ 2543.590156] Lustre: Skipped 1 previous similar message [ 2545.528894] LustreError: 153801:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2545.530962] LustreError: 153801:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2545.563902] Lustre: server umount lustre-MDT0000 complete [ 2546.761642] LustreError: 150302:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770449260 with bad export cookie 12960765527425454985 [ 2546.762809] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2546.765338] LustreError: 150302:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2548.703513] LustreError: 150317:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.703597] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2548.708542] LustreError: 150317:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 2548.716762] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2551.711476] LustreError: 150317:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2551.711480] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2552.892550] LustreError: 154002:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2552.895498] LustreError: 154002:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2552.934950] Lustre: server umount lustre-MDT0001 complete [ 2560.296642] LustreError: 154203:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2560.298740] LustreError: 154203:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2560.335660] Lustre: server umount lustre-OST0000 complete [ 2563.411937] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 2566.680276] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2578.546850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2578.642031] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2578.652627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2578.693245] Lustre: lustre-MDT0000: new disk, initializing [ 2578.716949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2578.724883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2580.065719] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2584.260411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2584.306993] Lustre: 156529:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2584.332563] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2584.335325] Lustre: Skipped 1 previous similar message [ 2584.373176] Lustre: lustre-MDT0001: new disk, initializing [ 2584.419425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2584.422746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2585.759890] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2588.796254] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2589.919524] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2592.112751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2592.246881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2592.249616] Lustre: Skipped 1 previous similar message [ 2594.066606] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2596.987222] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2597.874763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2597.878254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2597.908956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2598.162485] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2598.224210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2599.416771] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2599.489976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2602.977055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2602.977381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2602.980216] LustreError: Skipped 1 previous similar message [ 2602.984512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2602.987945] Lustre: Skipped 1 previous similar message [ 2606.189089] LustreError: 159462:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2606.191274] LustreError: 159462:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2606.232316] Lustre: server umount lustre-OST0000 complete [ 2608.098081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2608.103943] Lustre: Skipped 2 previous similar messages [ 2613.216620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2613.219745] Lustre: Skipped 3 previous similar messages [ 2613.664975] Lustre: server umount lustre-MDT0000 complete [ 2614.934949] LustreError: 156521:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770449328 with bad export cookie 12960765527425613801 [ 2614.938706] LustreError: 156521:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2614.939859] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2615.007231] LustreError: 159864:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2615.009719] LustreError: 159864:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2615.069703] Lustre: server umount lustre-MDT0001 complete [ 2617.863058] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:28:50 (1770449330) [ 2623.226844] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 2626.600418] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2637.695441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2637.795239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2637.806231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2637.807741] Lustre: Skipped 1 previous similar message [ 2637.839650] Lustre: lustre-MDT0000: new disk, initializing [ 2637.840857] Lustre: Skipped 1 previous similar message [ 2637.866235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2637.874854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2639.096451] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2643.012584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2643.056750] Lustre: 162838:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2643.162235] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2644.477751] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2647.583717] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2648.747657] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2651.055403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2653.166434] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2654.324451] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2654.355099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2656.223861] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2659.809374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2659.809575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2659.813138] LustreError: Skipped 1 previous similar message [ 2659.819737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2659.821598] Lustre: Skipped 1 previous similar message [ 2664.110102] LustreError: 165376:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2664.112101] LustreError: 165376:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2664.147143] Lustre: server umount lustre-OST0000 complete [ 2669.024062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2671.444750] Lustre: server umount lustre-MDT0000 complete [ 2672.697393] LustreError: 162829:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770449386 with bad export cookie 12960765527425614669 [ 2672.700658] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2672.703103] LustreError: 162829:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2672.810803] Lustre: server umount lustre-MDT0001 complete [ 2675.484129] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:29:48 (1770449388) [ 2680.496256] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 2683.674756] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2694.953306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2695.038338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2695.049047] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2695.050508] Lustre: Skipped 3 previous similar messages [ 2695.084392] Lustre: lustre-MDT0000: new disk, initializing [ 2695.086019] Lustre: Skipped 2 previous similar messages [ 2695.108397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.110783] Lustre: Skipped 2 previous similar messages [ 2695.115898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2695.118342] Lustre: Skipped 2 previous similar messages [ 2696.290518] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2700.382926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2700.424917] Lustre: 168754:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2700.519034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2701.744494] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2704.786115] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2705.954839] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2708.151441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2710.151087] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2710.183974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2713.121712] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2715.617113] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2715.619339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2715.621416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2715.622620] Lustre: Skipped 4 previous similar messages [ 2715.624181] Lustre: Skipped 3 previous similar messages [ 2721.005064] LustreError: 171291:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2721.007081] LustreError: 171291:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2721.041329] Lustre: server umount lustre-OST0000 complete [ 2728.403930] Lustre: server umount lustre-MDT0000 complete [ 2729.586142] LustreError: 168745:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770449443 with bad export cookie 12960765527425615740 [ 2729.589647] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2729.590475] LustreError: 168745:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2729.692472] Lustre: server umount lustre-MDT0001 complete [ 2732.838779] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2733.751907] Key type lgssc unregistered [ 2733.880286] LNet: 172225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2733.883067] LNetError: 172225:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2733.892302] LNet: Removed LNI 192.168.203.116@tcp [ 2734.158799] Key type .llcrypt unregistered [ 2734.159756] Key type ._llcrypt unregistered [ 2742.621261] Key type ._llcrypt registered [ 2742.622342] Key type .llcrypt registered [ 2742.658770] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2743.053656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2743.060707] alg: No test for adler32 (adler32-zlib) [ 2743.912327] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 2743.999102] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2745.583148] Key type lgssc registered [ 2745.930782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2748.383809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2748.393672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2749.544030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2750.747657] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2753.435426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2753.558240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2754.816330] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2756.610683] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2757.804029] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2759.984184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2760.098401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2761.880713] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2763.627946] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2774.438260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2788.831880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2788.834352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2788.837799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2789.856651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2789.856965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2789.857527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2792.814167] LustreError: 175863:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2792.854868] Lustre: server umount lustre-OST0000 complete [ 2794.463824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2794.465769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2794.469521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2794.471390] Lustre: Skipped 1 previous similar message [ 2800.096581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2800.099536] Lustre: Skipped 2 previous similar messages [ 2800.253642] LustreError: 176065:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2800.256089] LustreError: 176065:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2800.282576] Lustre: server umount lustre-MDT0000 complete [ 2801.447154] LustreError: 175212:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770449515 with bad export cookie 10382982487937560824 [ 2801.449731] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2801.451036] LustreError: 175212:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2801.516932] LustreError: 176266:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2801.518828] LustreError: 176266:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2801.560700] Lustre: server umount lustre-MDT0001 complete [ 2804.767251] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2805.677737] Key type lgssc unregistered [ 2805.788372] LNet: 176796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2805.791233] LNetError: 176796:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2805.804290] LNet: Removed LNI 192.168.203.116@tcp [ 2806.079470] Key type .llcrypt unregistered [ 2806.080910] Key type ._llcrypt unregistered [ 2814.752804] Key type ._llcrypt registered [ 2814.753882] Key type .llcrypt registered [ 2814.790659] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2815.189434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2815.225932] alg: No test for adler32 (adler32-zlib) [ 2816.072280] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 2816.143878] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2817.719104] Key type lgssc registered [ 2818.045828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2820.395461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2820.400752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2821.559410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2822.746385] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2825.412146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2825.530650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2826.725522] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2828.358472] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2829.474666] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2831.588186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2831.688466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2832.744368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2833.439395] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2835.215631] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2841.058087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2841.058432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2841.063440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2841.067060] Lustre: Skipped 1 previous similar message [ 2844.396099] LustreError: 180407:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2844.431632] Lustre: server umount lustre-OST0000 complete [ 2846.078655] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9b8bd0b78000.ost_server_uuid 50 [ 2846.527513] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9b8bd0b78000.ost_server_uuid in DISCONN state after 0 sec [ 2847.039339] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2861.088876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2861.089808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2861.092307] Lustre: Skipped 2 previous similar messages [ 2861.095413] Lustre: Skipped 2 previous similar messages [ 2863.099915] LustreError: 180803:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2863.101609] LustreError: 180803:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2863.127789] Lustre: server umount lustre-MDT0000 complete [ 2864.336241] LustreError: 178149:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770449578 with bad export cookie 11136732847524610553 [ 2864.338997] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2864.340372] LustreError: 178149:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2864.406561] LustreError: 181005:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2864.410264] LustreError: 181005:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2864.465872] Lustre: server umount lustre-MDT0001 complete [ 2867.046928] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:33:00 (1770449580) [ 2869.070197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2869.230838] LustreError: 181673:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.236181] LustreError: 181673:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 2869.251419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2870.458751] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2873.038039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2874.323110] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2876.020785] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2877.168289] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2879.294673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2879.403682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2879.406239] Lustre: Skipped 1 previous similar message [ 2880.425782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2881.142702] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2882.717805] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2884.736858] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2884.801498] Lustre: lustre-OST0001: new disk, initializing [ 2884.803327] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2886.460693] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2889.319227] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2890.209699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2890.211403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2890.213866] Lustre: Skipped 1 previous similar message [ 2890.217184] Lustre: Skipped 1 previous similar message [ 2894.318762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2894.320534] Lustre: Skipped 1 previous similar message [ 2894.325138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2894.328880] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2894.355639] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2896.236089] LustreError: 184857:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2896.238978] LustreError: 184857:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2896.254171] Lustre: server umount lustre-OST0000 complete [ 2898.001997] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2898.061376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2899.217959] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2899.286092] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2899.427429] LustreError: 183265:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2899.431904] LustreError: 183265:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 2904.544763] LustreError: 183263:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2904.549208] LustreError: 183263:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 2909.664699] LustreError: 184868:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2909.665671] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.665679] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2909.666177] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2909.666180] Lustre: Skipped 1 previous similar message [ 2909.668033] LustreError: 184868:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 2909.670895] Lustre: Skipped 1 previous similar message [ 2914.785215] LustreError: 184858:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2914.791040] LustreError: 184858:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 2915.316083] LustreError: 185569:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2915.317888] LustreError: 185569:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2915.350818] Lustre: server umount lustre-OST0001 complete [ 2919.391831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2919.394036] LustreError: Skipped 1 previous similar message [ 2919.395199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.399669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2919.402103] Lustre: Skipped 3 previous similar messages [ 2922.641432] Lustre: server umount lustre-MDT0000 complete [ 2923.732343] LustreError: 183286:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770449637 with bad export cookie 11136732847524611610 [ 2923.735434] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2923.735660] LustreError: 183286:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2923.791116] LustreError: 185971:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2923.793505] LustreError: 185971:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2923.840564] Lustre: server umount lustre-MDT0001 complete [ 2930.845336] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:34:03 (1770449643) [ 2932.906203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2932.996384] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2933.008148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2933.077915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2933.079942] Lustre: Skipped 1 previous similar message [ 2934.283468] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2936.989481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2937.021722] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2937.033831] Lustre: 187437:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2938.374341] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2940.099243] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2941.290362] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2943.515908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2943.577843] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2945.474817] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2947.342415] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2949.566786] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2949.590170] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2949.652047] LustreError: 189059:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2949.690620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 2949.722866] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2949.725780] Lustre: Skipped 2 previous similar messages [ 2951.663677] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2953.516061] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2954.722681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2954.725223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2954.727169] LustreError: Skipped 1 previous similar message [ 2954.728369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2954.728376] Lustre: Skipped 1 previous similar message [ 2954.733966] Lustre: Skipped 2 previous similar messages [ 2960.366199] LustreError: 190615:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2960.368494] LustreError: 190615:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2960.386659] Lustre: server umount lustre-OST0000 complete [ 2969.368060] LustreError: 189060:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2969.372380] LustreError: 189060:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 2975.203876] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2975.205539] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2975.208932] Lustre: Skipped 1 previous similar message [ 2977.040192] Lustre: server umount lustre-OST0001 complete [ 2984.340591] Lustre: server umount lustre-MDT0000 complete [ 2985.498354] LustreError: 188167:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770449699 with bad export cookie 11136732847524612968 [ 2985.500787] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2985.501996] LustreError: 188167:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2985.630561] Lustre: server umount lustre-MDT0001 complete [ 2993.012503] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:35:06 (1770449706) [ 2997.869987] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 3000.814900] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3011.819063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3011.896652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3011.906350] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3011.944308] Lustre: lustre-MDT0000: new disk, initializing [ 3011.966809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3011.972659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3013.222615] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3017.137959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3017.177318] Lustre: 195008:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3017.198427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3017.200510] Lustre: Skipped 1 previous similar message [ 3017.233815] Lustre: lustre-MDT0001: new disk, initializing [ 3017.269707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3017.272856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3018.459235] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3021.359437] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3022.431080] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3024.485730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3024.571271] Lustre: lustre-OST0000: new disk, initializing [ 3024.572845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3026.312751] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3027.569659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3027.572428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3027.597322] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3029.231792] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3030.519976] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3030.586203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3031.851816] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3031.915820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3033.057207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3033.057956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3033.058401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3033.058404] Lustre: Skipped 9 previous similar messages [ 3033.059355] LustreError: Skipped 3 previous similar messages [ 3033.066071] Lustre: Skipped 3 previous similar messages [ 3038.574096] LustreError: 197939:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3038.576629] LustreError: 197939:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3038.610617] Lustre: server umount lustre-OST0000 complete [ 3054.047116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3054.105406] Lustre: server umount lustre-MDT0000 complete [ 3055.271078] LustreError: 195000:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770449769 with bad export cookie 11136732847524614431 [ 3055.273148] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3055.274785] LustreError: 195000:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3055.380575] Lustre: server umount lustre-MDT0001 complete [ 3058.079383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3058.225361] LustreError: 198818:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3058.231782] LustreError: 198818:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 6 previous similar messages [ 3059.503826] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3062.082675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3063.424648] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3065.102629] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3066.346785] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3068.471159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3070.429829] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3072.143412] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3073.374285] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3084.640847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec [ 3085.925724] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3085.997699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3088.351911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3088.356149] LustreError: Skipped 1 previous similar message [ 3092.686052] Lustre: server umount lustre-OST0000 complete [ 3094.602810] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3095.942901] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3119.566307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3119.689984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3119.693685] Lustre: Skipped 5 previous similar messages [ 3121.680165] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3123.459672] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3123.988100] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3127.081300] LustreError: 203263:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3127.082969] LustreError: 203263:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3127.111278] Lustre: server umount lustre-OST0000 complete [ 3128.799841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3128.802503] LustreError: Skipped 1 previous similar message [ 3128.803880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.807628] Lustre: Skipped 4 previous similar messages [ 3128.809551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3128.811727] Lustre: Skipped 13 previous similar messages [ 3130.395249] Lustre: server umount lustre-MDT0000 complete [ 3131.726279] LustreError: 198798:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770449845 with bad export cookie 11136732847524615299 [ 3131.729207] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3131.729337] LustreError: 198798:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3131.833995] Lustre: server umount lustre-MDT0001 complete [ 3134.475412] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:37:27 (1770449847) [ 3136.579635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3136.716768] LustreError: 204338:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3137.903068] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3140.619295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3141.922608] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3143.591584] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3144.713664] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3146.875654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3148.710349] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3150.326668] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3151.642425] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3151.707065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3152.918684] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3152.984217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3155.029643] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3155.096593] Lustre: lustre-OST0001: new disk, initializing [ 3155.098408] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3156.902723] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3159.778433] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3160.576341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3160.579214] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3160.606660] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3161.006569] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3161.071251] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3162.258338] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3162.321771] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3168.955788] Lustre: server umount lustre-OST0001 complete [ 3170.694497] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3170.752039] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3171.927733] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3171.989862] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3195.420184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3197.241994] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3198.868768] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3199.428219] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3206.134187] Lustre: server umount lustre-OST0001 complete [ 3219.935766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3219.938650] LustreError: Skipped 2 previous similar messages [ 3224.138458] Lustre: server umount lustre-OST0000 complete [ 3231.511838] Lustre: server umount lustre-MDT0000 complete [ 3232.692055] LustreError: 204319:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770449946 with bad export cookie 11136732847524616230 [ 3232.694367] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3232.695922] LustreError: 204319:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3238.822464] Lustre: server umount lustre-MDT0001 complete [ 3246.172450] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:39:19 (1770449959) [ 3248.194075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3248.286038] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3248.297462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3248.364123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3248.366582] Lustre: Skipped 5 previous similar messages [ 3249.517938] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3252.246203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3252.277137] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3252.287328] Lustre: 212275:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3253.755166] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3255.539855] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3256.739571] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3258.981748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3259.049381] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3260.869569] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3262.664554] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3266.421455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3268.272062] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3270.048192] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3272.648068] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 3274.007425] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3274.075214] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3275.352921] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3275.422614] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3276.629870] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b8bc3839800.ost_server_uuid 50 [ 3277.070050] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b8bc3839800.ost_server_uuid in FULL state after 0 sec [ 3277.376669] Lustre: Permanently deactivating lustre-OST0001 [ 3277.381373] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3278.181647] Lustre: Permanently reactivating lustre-OST0001 [ 3280.867866] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3280.871181] Lustre: Skipped 7 previous similar messages [ 3280.872946] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3280.875250] Lustre: Skipped 19 previous similar messages [ 3285.230083] LustreError: 216244:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3285.232128] LustreError: 216244:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 3285.249922] Lustre: server umount lustre-OST0001 complete [ 3285.987454] LustreError: 214413:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3285.991923] LustreError: 214413:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 23 previous similar messages [ 3292.620783] Lustre: server umount lustre-OST0000 complete [ 3300.000397] Lustre: server umount lustre-MDT0000 complete [ 3301.265178] LustreError: 212267:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770450015 with bad export cookie 11136732847524617490 [ 3301.267846] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3301.268378] LustreError: 212267:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3301.383694] Lustre: server umount lustre-MDT0001 complete [ 3308.746122] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:40:21 (1770450021) [ 3311.193104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3311.302943] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3311.314746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3311.317713] Lustre: Skipped 5 previous similar messages [ 3312.706451] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3315.502027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3315.540659] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3315.542272] Lustre: Skipped 1 previous similar message [ 3315.551515] Lustre: 218371:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3316.990195] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3318.785122] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3320.001231] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3322.166860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3322.240383] Lustre: Permanently deactivating lustre-OST0000 [ 3322.244806] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3324.101419] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3325.422627] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3325.809859] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3328.061503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3329.952514] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3331.815642] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3333.147625] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3337.399765] Lustre: Permanently reactivating lustre-OST0000 [ 3342.837585] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3342.841347] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3342.846456] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3342.865666] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3342.877648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3348.486595] Lustre: server umount lustre-OST0001 complete [ 3353.056290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3353.060123] LustreError: Skipped 7 previous similar messages [ 3364.319092] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3364.376168] Lustre: server umount lustre-OST0000 complete [ 3371.874906] Lustre: server umount lustre-MDT0000 complete [ 3373.148163] LustreError: 218363:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770450087 with bad export cookie 11136732847524619128 [ 3373.152181] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3373.153586] LustreError: 218363:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3373.277112] Lustre: server umount lustre-MDT0001 complete [ 3376.991849] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3378.049095] Key type lgssc unregistered [ 3378.189434] LNet: 222751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3378.192261] LNetError: 222751:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3378.201298] LNet: Removed LNI 192.168.203.116@tcp [ 3378.527342] Key type .llcrypt unregistered [ 3378.528876] Key type ._llcrypt unregistered [ 3387.269694] Key type ._llcrypt registered [ 3387.270730] Key type .llcrypt registered [ 3387.311739] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3387.795295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3387.834174] alg: No test for adler32 (adler32-zlib) [ 3388.703571] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 3388.799936] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3390.391210] Key type lgssc registered [ 3390.846569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3394.307857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3394.313429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3395.474297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3396.729918] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3399.508372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3399.544902] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3399.555448] Lustre: 224164:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3399.562587] Lustre: Permanently deactivating lustre-MDT0001 [ 3399.570124] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3399.692244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3400.949409] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3402.739293] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3403.977419] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3404.268173] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3404.270598] LustreError: 224206:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3404.275967] LustreError: 224206:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3404.279539] LustreError: 224206:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 3406.251739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3406.366546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3408.302581] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3409.444047] LustreError: 226289:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3409.449121] LustreError: 226289:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 3410.107793] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3412.445887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3412.511174] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3412.890489] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3414.456301] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3416.284695] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3420.072390] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3420.072805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3421.911610] Lustre: Permanently reactivating lustre-MDT0001 [ 3421.918398] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3421.921520] Lustre: Skipped 2 previous similar messages [ 3423.581837] Lustre: lustre-MDT0001: Client 0eb92e99-719f-4683-822b-88ce98e55835 (at 192.168.203.16@tcp) reconnecting [ 3425.171919] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3425.703650] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3427.379580] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3427.455219] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3428.790516] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3428.866625] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3429.723631] Lustre: Permanently deactivating lustre-MDT0001 [ 3429.731436] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3429.733802] Lustre: Skipped 2 previous similar messages [ 3435.009532] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3439.073746] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3439.076124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3439.081535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3442.657286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3442.658436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3442.660968] Lustre: Skipped 2 previous similar messages [ 3442.663076] Lustre: Skipped 2 previous similar messages [ 3444.541957] LustreError: 228232:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3444.569617] Lustre: server umount lustre-MDT0000 complete [ 3445.790812] LustreError: 224155:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770450159 with bad export cookie 12403075219885220592 [ 3445.794719] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3445.795638] LustreError: 224155:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3445.868604] LustreError: 228434:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3445.871757] LustreError: 228434:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3445.919718] Lustre: server umount lustre-MDT0001 complete [ 3453.230089] LustreError: 228634:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3453.232367] LustreError: 228634:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3453.248142] Lustre: server umount lustre-OST0000 complete [ 3456.874873] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:42:49 (1770450169) [ 3460.453964] LustreError: 229638:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3460.455894] LustreError: 229638:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3460.492855] Lustre: server umount lustre-OST0001 complete [ 3462.339821] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 3465.061361] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3476.852550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3476.945904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3476.948860] Lustre: Skipped 2 previous similar messages [ 3476.958252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3477.001886] Lustre: lustre-MDT0000: new disk, initializing [ 3477.030155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3477.038678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3478.377970] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3482.654631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3482.696425] Lustre: 231807:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3482.719667] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3482.721338] Lustre: Skipped 1 previous similar message [ 3482.757592] Lustre: lustre-MDT0001: new disk, initializing [ 3482.805182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3482.809341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3484.174478] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3487.292159] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3488.541492] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3490.823287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3490.926655] Lustre: lustre-OST0000: new disk, initializing [ 3490.928576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3490.959150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3490.961305] Lustre: Skipped 1 previous similar message [ 3492.871385] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3495.979045] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3499.513556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3499.517125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3499.548483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3501.852655] LustreError: 233137:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3506.039758] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3506.102471] Lustre: lustre-OST0001: new disk, initializing [ 3506.105181] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3506.569122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3506.572289] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3506.599729] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3508.055483] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3511.126198] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3511.700432] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3511.943143] LustreError: 233137:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3511.952990] LustreError: 233137:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3522.047061] LustreError: 233137:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3526.436648] Lustre: lustre-OST0001: Not available for connect from 192.168.203.16@tcp (stopping) [ 3526.439731] Lustre: Skipped 1 previous similar message [ 3527.141160] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3527.142667] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3527.143748] LustreError: Skipped 1 previous similar message [ 3527.151918] Lustre: Skipped 1 previous similar message [ 3529.004087] LustreError: 235315:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3529.006909] LustreError: 235315:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3529.026399] Lustre: server umount lustre-OST0001 complete [ 3532.261744] LustreError: 233534:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3532.266313] LustreError: 233534:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 3537.379660] LustreError: 234158:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3537.388898] LustreError: 234158:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 3542.498657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3542.499820] LustreError: 234157:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3542.500298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3542.500307] Lustre: Skipped 2 previous similar messages [ 3542.502683] Lustre: Skipped 1 previous similar message [ 3542.511269] LustreError: 234157:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 3547.181123] LustreError: 235538:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3547.183819] LustreError: 235538:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3547.229355] Lustre: server umount lustre-OST0000 complete [ 3549.663750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3549.666131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3549.670306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3549.671929] Lustre: Skipped 1 previous similar message [ 3554.714709] Lustre: server umount lustre-MDT0000 complete [ 3555.946891] LustreError: 234552:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770450269 with bad export cookie 12403075219885223224 [ 3555.949948] LustreError: 234552:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3555.950336] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3562.096951] Lustre: server umount lustre-MDT0001 complete [ 3565.871059] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3566.900628] Key type lgssc unregistered [ 3567.033311] LNet: 236474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3567.035909] LNetError: 236474:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3567.042316] LNet: Removed LNI 192.168.203.116@tcp [ 3567.338384] Key type .llcrypt unregistered [ 3567.339538] Key type ._llcrypt unregistered [ 3581.342176] Key type ._llcrypt registered [ 3581.343261] Key type .llcrypt registered [ 3581.381873] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3581.755123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3581.803031] alg: No test for adler32 (adler32-zlib) [ 3582.679751] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 3582.774123] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3584.359153] Key type lgssc registered [ 3584.720502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3587.229772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3587.236194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3588.365658] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3588.373467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3588.442795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3589.864972] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3592.754910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3592.791042] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3592.804337] Lustre: 238638:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3592.917035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3594.199381] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3596.011653] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3597.259244] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3599.537876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3599.605405] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3599.672768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3601.522235] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3603.358148] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3604.495500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3650.529182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3650.531133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.531690] LustreError: Skipped 1 previous similar message [ 3650.537229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3650.541113] Lustre: Skipped 1 previous similar message [ 3654.065698] LustreError: 240990:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3654.115815] Lustre: server umount lustre-OST0000 complete [ 3655.658626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3659.757566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3659.869500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3661.747792] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3663.445753] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3670.116970] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3670.121520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3670.121598] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3670.127592] Lustre: Skipped 1 previous similar message [ 3701.218332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3701.219089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3701.222128] Lustre: Skipped 1 previous similar message [ 3701.224550] LustreError: Skipped 1 previous similar message [ 3701.225069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3701.230115] Lustre: Skipped 1 previous similar message [ 3702.700428] LustreError: 242651:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3702.703192] LustreError: 242651:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3702.753473] Lustre: server umount lustre-OST0000 complete [ 3705.824076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3705.826600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3705.831042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3710.083782] LustreError: 242855:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3710.085768] LustreError: 242855:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3710.119196] Lustre: server umount lustre-MDT0000 complete [ 3711.406731] LustreError: 238630:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770450425 with bad export cookie 16965740587342621170 [ 3711.411884] LustreError: 238630:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3711.412732] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3711.544852] Lustre: server umount lustre-MDT0001 complete [ 3715.115637] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3716.116171] Key type lgssc unregistered [ 3716.249364] LNet: 243586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3716.251858] LNetError: 243586:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3716.260377] LNet: Removed LNI 192.168.203.116@tcp [ 3716.567577] Key type .llcrypt unregistered [ 3716.568688] Key type ._llcrypt unregistered [ 3725.627280] Key type ._llcrypt registered [ 3725.629099] Key type .llcrypt registered [ 3725.665270] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3725.982370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3726.012040] alg: No test for adler32 (adler32-zlib) [ 3726.860103] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 3726.944082] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3728.535189] Key type lgssc registered [ 3728.995152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3731.674128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3731.681611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.839033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3734.227509] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3737.424617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3737.545143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3738.803420] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3740.663297] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3741.982642] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3744.363570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3744.479056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3746.566211] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3748.455248] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3757.803856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3768.296720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3768.297701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3768.300619] LustreError: Skipped 1 previous similar message [ 3768.304211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3768.307355] Lustre: Skipped 1 previous similar message [ 3772.337200] LustreError: 247993:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3772.400463] Lustre: server umount lustre-OST0000 complete [ 3778.529081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3778.529150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3778.529719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3778.537540] Lustre: Skipped 2 previous similar messages [ 3783.648576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3783.650664] Lustre: Skipped 2 previous similar messages [ 3788.255159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3788.301207] LustreError: 248195:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3788.303689] LustreError: 248195:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3788.331924] Lustre: server umount lustre-MDT0000 complete [ 3788.768648] LustreError: 244959:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3788.775613] LustreError: 244959:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 3789.748752] LustreError: 244938:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770450503 with bad export cookie 6962410521833717223 [ 3789.754147] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3789.819787] LustreError: 248398:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3789.822399] LustreError: 248398:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3789.879856] Lustre: server umount lustre-MDT0001 complete [ 3793.558395] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3794.649299] Key type lgssc unregistered [ 3794.783426] LNet: 248929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3794.786522] LNetError: 248929:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3794.795448] LNet: Removed LNI 192.168.203.116@tcp [ 3795.087663] Key type .llcrypt unregistered [ 3795.089127] Key type ._llcrypt unregistered [ 3802.478733] Key type ._llcrypt registered [ 3802.479968] Key type .llcrypt registered [ 3802.515302] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3802.943581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3802.950907] alg: No test for adler32 (adler32-zlib) [ 3803.799439] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 3803.880869] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3805.463159] Key type lgssc registered [ 3805.856417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3808.511873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3808.517263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3809.670481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3810.911115] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3813.757194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3813.877767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3815.151331] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3816.940504] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3818.174154] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3820.427143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3820.544551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3822.398805] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3824.224635] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3835.943196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3844.063886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3844.065759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3844.068992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3846.115240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3846.116853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3846.116993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3849.389943] LustreError: 252517:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3849.428502] Lustre: server umount lustre-OST0000 complete [ 3851.232621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3851.236974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3851.239262] Lustre: Skipped 2 previous similar messages [ 3856.352123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3856.353784] Lustre: Skipped 1 previous similar message [ 3856.833415] LustreError: 252719:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3856.835392] LustreError: 252719:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3856.862164] Lustre: server umount lustre-MDT0000 complete [ 3858.090120] LustreError: 250088:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770450572 with bad export cookie 11670661750003833704 [ 3858.094177] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3858.094874] LustreError: 250088:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3858.163879] LustreError: 252923:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3858.165886] LustreError: 252923:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3858.215910] Lustre: server umount lustre-MDT0001 complete [ 3861.916952] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3862.944305] Key type lgssc unregistered [ 3863.075390] LNet: 253455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3863.077762] LNetError: 253455:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3863.085465] LNet: Removed LNI 192.168.203.116@tcp [ 3863.409201] Key type .llcrypt unregistered [ 3863.410289] Key type ._llcrypt unregistered [ 3872.332494] Key type ._llcrypt registered [ 3872.333576] Key type .llcrypt registered [ 3872.369786] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3872.708549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3872.763661] alg: No test for adler32 (adler32-zlib) [ 3873.618662] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 3873.699467] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3875.271102] Key type lgssc registered [ 3875.596660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3877.959762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3877.969338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3879.115522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3880.381420] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3883.224740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3883.343813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3884.626715] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3886.391392] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3887.563858] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3889.797667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3889.912209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3891.807781] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3893.660530] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3904.231287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3914.723069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3914.723554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3914.726073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3914.732466] Lustre: Skipped 1 previous similar message [ 3917.815191] LustreError: 257865:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3917.861313] Lustre: server umount lustre-OST0000 complete [ 3919.840492] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3919.844614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3919.846275] Lustre: Skipped 1 previous similar message [ 3924.448544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3924.453887] LustreError: Skipped 1 previous similar message [ 3924.456973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3924.466374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3925.517960] LustreError: 258067:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3925.522241] LustreError: 258067:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3925.572603] Lustre: server umount lustre-MDT0000 complete [ 3927.106074] LustreError: 258069:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770450641 with bad export cookie 12819842960486583587 [ 3927.109296] LustreError: 258069:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3927.109490] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3927.182446] LustreError: 258270:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3927.184368] LustreError: 258270:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3927.234665] Lustre: server umount lustre-MDT0001 complete [ 3930.951954] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3931.951659] Key type lgssc unregistered [ 3932.080433] LNet: 258800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3932.084201] LNetError: 258800:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3932.092310] LNet: Removed LNI 192.168.203.116@tcp [ 3932.377570] Key type .llcrypt unregistered [ 3932.379164] Key type ._llcrypt unregistered [ 3939.305159] Key type ._llcrypt registered [ 3939.306392] Key type .llcrypt registered [ 3939.341283] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3939.659561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3939.708061] alg: No test for adler32 (adler32-zlib) [ 3940.556797] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 3940.639340] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3942.223136] Key type lgssc registered [ 3942.576381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3945.137328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3945.143418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3946.305820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3947.603119] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3950.441265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3950.564288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3951.952756] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3953.648192] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3955.038969] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3958.005947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3958.152562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3960.512732] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3962.531545] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3971.493781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3976.673161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3976.674327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3976.674878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3976.681384] Lustre: Skipped 1 previous similar message [ 3978.351277] LustreError: 262422:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3978.394369] Lustre: server umount lustre-OST0000 complete [ 3981.279903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3981.282087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3981.285818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3981.287609] Lustre: Skipped 1 previous similar message [ 3981.692864] LustreError: 262623:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3981.694738] LustreError: 262623:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3981.722227] Lustre: server umount lustre-MDT0000 complete [ 3981.793744] LustreError: 259986:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3981.798520] LustreError: 259986:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 3982.930844] LustreError: 260735:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770450696 with bad export cookie 469990022138933526 [ 3982.934422] LustreError: 260735:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3982.935451] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3983.007244] LustreError: 262823:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3983.009637] LustreError: 262823:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3983.077235] Lustre: server umount lustre-MDT0001 complete [ 3986.769466] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3987.764462] Key type lgssc unregistered [ 3987.900345] LNet: 263356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3987.903932] LNetError: 263356:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3987.912298] LNet: Removed LNI 192.168.203.116@tcp [ 3988.215198] Key type .llcrypt unregistered [ 3988.216280] Key type ._llcrypt unregistered [ 3991.882116] Key type ._llcrypt registered [ 3991.883431] Key type .llcrypt registered [ 3991.925742] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3997.734806] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 4003.795914] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4004.130367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4004.144258] alg: No test for adler32 (adler32-zlib) [ 4005.012797] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 4005.105206] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4006.687209] Key type lgssc registered [ 4007.042584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4018.426995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4018.435055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4019.544129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4019.561735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4019.607303] Lustre: lustre-MDT0000: new disk, initializing [ 4019.632489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4019.639690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4021.119689] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4025.917410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4025.987200] Lustre: 266827:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4026.037140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4026.042661] Lustre: Skipped 1 previous similar message [ 4026.097928] Lustre: lustre-MDT0001: new disk, initializing [ 4026.154919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4026.178588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4026.185125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4027.709372] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4030.910610] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4032.089919] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4034.398682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4034.513904] Lustre: lustre-OST0000: new disk, initializing [ 4034.516319] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4034.550847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4035.577870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4035.581361] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4035.613466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4036.635065] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4039.723390] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4041.059346] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4041.125956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4042.360387] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4042.435851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4045.102104] LustreError: 269757:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4045.141642] Lustre: server umount lustre-OST0000 complete [ 4046.815709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4046.818253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4046.822772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4050.912483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4050.912899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4056.032372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4056.034315] Lustre: Skipped 2 previous similar messages [ 4056.037107] LustreError: 269958:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4056.039493] LustreError: 269958:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4056.070501] Lustre: server umount lustre-MDT0000 complete [ 4057.331929] LustreError: 266817:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770450771 with bad export cookie 14331795241682838541 [ 4057.335510] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4057.335924] LustreError: 266817:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4067.812215] LustreError: 270161:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4067.814469] LustreError: 270161:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4067.868369] Lustre: server umount lustre-MDT0001 complete [ 4070.736420] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 02:53:03 (1770450783) [ 4073.023704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4073.199697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4074.522719] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4077.374299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4077.514501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4078.902317] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4080.633071] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4081.887632] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4084.193080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4086.255766] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4088.125548] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4103.136388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4103.138607] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4103.142439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4104.162750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4107.949506] LustreError: 273093:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4107.951944] LustreError: 273093:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4107.990063] Lustre: server umount lustre-OST0000 complete [ 4109.281407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.282069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4109.285104] Lustre: Skipped 2 previous similar messages [ 4109.287383] Lustre: Skipped 2 previous similar messages [ 4115.455871] LustreError: 273296:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4115.457675] LustreError: 273296:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4115.484137] Lustre: server umount lustre-MDT0000 complete [ 4116.706395] LustreError: 272445:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770450830 with bad export cookie 14331795241682839409 [ 4116.711282] LustreError: 271550:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4116.712039] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4116.712359] LustreError: 272445:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4116.851335] Lustre: server umount lustre-MDT0001 complete [ 4120.446703] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4121.419748] Key type lgssc unregistered [ 4121.545367] LNet: 274030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4121.547857] LNetError: 274030:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4121.555358] LNet: Removed LNI 192.168.203.116@tcp [ 4121.854158] Key type .llcrypt unregistered [ 4121.855904] Key type ._llcrypt unregistered [ 4130.722684] Key type ._llcrypt registered [ 4130.723789] Key type .llcrypt registered [ 4130.759366] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 4136.924844] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4137.222304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4137.229179] alg: No test for adler32 (adler32-zlib) [ 4138.091977] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 4138.177539] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4139.759164] Key type lgssc registered [ 4140.128582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4154.680453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4154.686877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4155.782386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4155.795148] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4155.838605] Lustre: lustre-MDT0000: new disk, initializing [ 4155.869400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4155.878798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4157.402717] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4161.677934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4161.722286] Lustre: 277790:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4161.748988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4161.751033] Lustre: Skipped 1 previous similar message [ 4161.797577] Lustre: lustre-MDT0001: new disk, initializing [ 4161.836140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4161.849692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4161.853499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4163.167574] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4166.367250] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4167.635363] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4170.027904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4170.154460] Lustre: lustre-OST2710: new disk, initializing [ 4170.157233] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4170.193663] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4171.776480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4171.780574] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4171.828520] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4172.218078] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4175.418582] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4177.763981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4177.825910] Lustre: lustre-OST03e8: new disk, initializing [ 4177.827854] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4177.860781] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4179.926781] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4181.801575] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4184.068890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4184.073188] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4184.108202] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4184.499420] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4184.576104] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4185.986572] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4186.064379] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4187.395330] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4187.479807] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4188.847739] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4188.926729] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4192.735866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4192.739101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4192.743654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4194.272868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4194.275937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.279755] Lustre: Skipped 1 previous similar message [ 4196.096376] LustreError: 282120:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4196.126745] Lustre: server umount lustre-MDT0000 complete [ 4197.385091] LustreError: 279529:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770450911 with bad export cookie 5123428973061080454 [ 4197.387950] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4197.390203] LustreError: 279529:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4197.471249] LustreError: 282329:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4197.474557] LustreError: 282329:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4197.528650] Lustre: server umount lustre-MDT0001 complete [ 4208.564083] LustreError: 282531:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4208.566492] LustreError: 282531:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4208.582881] Lustre: server umount lustre-OST2710 complete [ 4220.018871] LustreError: 282733:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4220.022061] LustreError: 282733:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4220.069809] Lustre: server umount lustre-OST03e8 complete [ 4225.879731] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 4229.048602] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4240.563510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4240.650878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4240.663723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4240.700365] Lustre: lustre-MDT0000: new disk, initializing [ 4240.724510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4240.730686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4242.009910] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4246.147511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4246.191657] Lustre: 285513:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4246.290222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4247.602740] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4250.695767] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4251.858686] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4254.116970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4254.223574] Lustre: lustre-OST0000: new disk, initializing [ 4254.225476] Lustre: Skipped 1 previous similar message [ 4254.227793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4254.230727] Lustre: Skipped 2 previous similar messages [ 4254.258927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4254.261282] Lustre: Skipped 1 previous similar message [ 4254.908871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4254.911497] Lustre: Skipped 1 previous similar message [ 4254.913327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4254.978173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4256.128290] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4259.184494] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4260.492816] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4260.564705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4261.915974] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4262.012641] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4265.441539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4265.443964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4265.446962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4265.447910] Lustre: Skipped 2 previous similar messages [ 4265.449966] Lustre: Skipped 3 previous similar messages [ 4268.718108] LustreError: 288436:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4268.720534] LustreError: 288436:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4268.762427] Lustre: server umount lustre-OST0000 complete [ 4270.560729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4270.561657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4270.564149] Lustre: Skipped 1 previous similar message [ 4270.568625] Lustre: Skipped 2 previous similar messages [ 4275.680554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4275.683224] Lustre: Skipped 1 previous similar message [ 4284.383244] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4284.437666] LustreError: 288637:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4284.440778] LustreError: 288637:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4284.483322] Lustre: server umount lustre-MDT0000 complete [ 4285.778571] LustreError: 285503:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770450999 with bad export cookie 5123428973061082106 [ 4285.781939] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4285.782174] LustreError: 285503:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4285.916782] Lustre: server umount lustre-MDT0001 complete [ 4288.594354] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 02:56:41 (1770451001) [ 4289.081191] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4289.642920] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 02:56:42 (1770451002) [ 4290.662891] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4299.817370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4299.910384] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4299.919254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4299.983506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4301.255274] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4304.275350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4304.324285] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4304.332931] Lustre: 290934:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4305.675127] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4307.632551] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4308.819608] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4322.523519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4322.627121] Lustre: Denying initial registration attempt from nid 192.168.203.116@tcp, specified as failover [ 4322.633664] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4322.639188] LustreError: 292546:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -99 [ 4322.643902] LustreError: 292546:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4322.647806] LustreError: 292546:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4322.679299] Lustre: server umount lustre-OST0000 complete [ 4322.681312] LustreError: 292546:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4326.370040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4326.371155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4326.377315] Lustre: Skipped 1 previous similar message [ 4326.381704] Lustre: Skipped 2 previous similar messages [ 4331.197066] LustreError: 292913:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4331.199399] LustreError: 292913:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4331.222046] Lustre: server umount lustre-MDT0000 complete [ 4332.440506] LustreError: 291656:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451046 with bad export cookie 5123428973061082918 [ 4332.443731] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4332.443911] LustreError: 291656:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4332.566850] Lustre: server umount lustre-MDT0001 complete [ 4336.154922] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4337.199971] Key type lgssc unregistered [ 4337.328398] LNet: 293648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4337.332256] LNetError: 293648:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4337.340343] LNet: Removed LNI 192.168.203.116@tcp [ 4337.640935] Key type .llcrypt unregistered [ 4337.642579] Key type ._llcrypt unregistered [ 4341.342819] Key type ._llcrypt registered [ 4341.344869] Key type .llcrypt registered [ 4341.382449] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4341.819456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4341.828849] alg: No test for adler32 (adler32-zlib) [ 4342.684252] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 4342.770232] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4344.351120] Key type lgssc registered [ 4344.718818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4381.020094] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4384.086808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4384.093609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4385.190192] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4385.199665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4385.261124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4386.644983] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4389.482473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4389.516134] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4389.528331] Lustre: 296356:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4389.635439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4390.948735] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4392.718861] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4393.938170] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4407.170913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4407.299582] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4407.396385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4409.407816] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4411.151391] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4411.866115] LustreError: 298623:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4411.905443] Lustre: server umount lustre-OST0000 complete [ 4413.935057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4413.935894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4413.938235] Lustre: Skipped 1 previous similar message [ 4419.040718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4419.043233] Lustre: Skipped 2 previous similar messages [ 4419.267681] LustreError: 298823:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4419.270377] LustreError: 298823:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4419.290460] Lustre: server umount lustre-MDT0000 complete [ 4420.550633] LustreError: 296348:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451134 with bad export cookie 7000546512271150368 [ 4420.555110] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4420.556190] LustreError: 296348:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4420.635695] LustreError: 299028:0:(obd_class.h:479:obd_check_dev()) Device 7 not setup [ 4420.637854] LustreError: 299028:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4420.693563] Lustre: server umount lustre-MDT0001 complete [ 4424.173150] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4425.097661] Key type lgssc unregistered [ 4425.216369] LNet: 299559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4425.219754] LNetError: 299559:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4425.230280] LNet: Removed LNI 192.168.203.116@tcp [ 4425.514267] Key type .llcrypt unregistered [ 4425.515350] Key type ._llcrypt unregistered [ 4434.623825] Key type ._llcrypt registered [ 4434.624850] Key type .llcrypt registered [ 4434.657640] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4435.031218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4435.067178] alg: No test for adler32 (adler32-zlib) [ 4435.919916] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 4436.000071] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4437.591136] Key type lgssc registered [ 4437.985578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4440.635829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4440.642494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4441.811367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4443.366458] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4446.159963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4446.278639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4447.532487] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4449.267372] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4450.422389] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4452.659918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4452.791578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4454.600862] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4456.304363] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4476.895767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4476.899090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4476.902803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4477.408929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4477.409673] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4477.410754] Lustre: Skipped 1 previous similar message [ 4477.419624] Lustre: Skipped 1 previous similar message [ 4480.122768] LustreError: 303196:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4480.152708] Lustre: server umount lustre-MDT0000 complete [ 4481.341356] LustreError: 300912:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451195 with bad export cookie 6847811229646610913 [ 4481.345900] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4482.327899] LustreError: 300928:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4482.333251] LustreError: 300928:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 4482.334943] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [ 4482.338060] Lustre: Skipped 2 previous similar messages [ 4482.528929] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4486.559759] LustreError: 300927:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4486.559982] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4486.564063] LustreError: 300927:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 4486.565840] Lustre: Skipped 1 previous similar message [ 4487.485313] LustreError: 303399:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4487.487141] LustreError: 303399:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4487.530665] Lustre: server umount lustre-MDT0001 complete [ 4489.652146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4492.510427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.079710] LustreError: 300244:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b1f93d78 x1856452585086592/t0(0) o250->MGC192.168.203.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4506.190372] LustreError: 302535:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4506.195098] LustreError: 302535:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 4506.214823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4506.239294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4507.600075] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4510.391247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4510.525275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4511.587666] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4511.792345] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4513.523848] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4516.708954] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4519.112196] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4520.239236] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4520.753878] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4521.953826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4521.955358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4521.958214] Lustre: Skipped 1 previous similar message [ 4521.960201] Lustre: Skipped 1 previous similar message [ 4521.962721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4532.192797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4532.194777] Lustre: Skipped 4 previous similar messages [ 4536.287151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4536.310887] LustreError: 305701:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4536.314048] LustreError: 305701:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4536.356620] Lustre: server umount lustre-OST0000 complete [ 4541.408086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4541.410111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4543.749780] LustreError: 305905:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4543.752695] LustreError: 305905:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4543.781664] Lustre: server umount lustre-MDT0000 complete [ 4545.128074] LustreError: 302548:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451259 with bad export cookie 6847811229646618067 [ 4545.131692] LustreError: 302548:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4545.132150] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4545.243647] Lustre: server umount lustre-MDT0001 complete [ 4548.701430] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4549.775498] Key type lgssc unregistered [ 4549.912418] LNet: 306650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4549.915268] LNetError: 306650:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4549.923449] LNet: Removed LNI 192.168.203.116@tcp [ 4550.225539] Key type .llcrypt unregistered [ 4550.226586] Key type ._llcrypt unregistered [ 4558.853834] Key type ._llcrypt registered [ 4558.854807] Key type .llcrypt registered [ 4558.893184] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4559.301798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4559.310245] alg: No test for adler32 (adler32-zlib) [ 4560.155508] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 4560.233327] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4561.815131] Key type lgssc registered [ 4562.154748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4564.682892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4564.690581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4565.854488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4567.235834] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4570.344909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4570.459626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4571.800776] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4573.538915] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4574.763432] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4580.001238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4580.005445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4580.007383] Lustre: Skipped 1 previous similar message [ 4582.014051] LustreError: 309409:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4582.041505] Lustre: server umount lustre-MDT0000 complete [ 4583.305731] LustreError: 308003:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451297 with bad export cookie 12061664876482774515 [ 4583.309966] LustreError: 308003:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4583.311085] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4583.392549] LustreError: 309612:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4583.394812] LustreError: 309612:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4583.451538] Lustre: server umount lustre-MDT0001 complete [ 4588.417292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4588.522819] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4588.537483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4588.623038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4589.891898] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4593.120375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4593.161466] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4593.172942] Lustre: 310380:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4593.262296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4594.698318] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4597.114312] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4598.567821] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4601.689690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4601.805446] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4601.809518] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4601.813685] LustreError: 312035:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -2 [ 4601.816379] LustreError: 312035:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4601.818526] LustreError: 312035:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4601.845745] Lustre: server umount lustre-OST0000 complete [ 4601.847780] LustreError: 312035:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4604.755327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4604.827592] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4604.885702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4606.743578] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4608.530720] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4610.079991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4611.062909] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4611.094651] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4611.125353] Lustre: lustre-OST0001: new disk, initializing [ 4611.127296] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4612.884522] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4614.540716] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4615.600217] LustreError: 314138:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4615.605254] LustreError: 314138:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4615.632693] Lustre: server umount lustre-OST0001 complete [ 4616.197598] LustreError: 312499:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4616.203778] LustreError: 312499:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 4620.256391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4620.256563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4620.259178] LustreError: Skipped 1 previous similar message [ 4620.259878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4620.264358] Lustre: Skipped 2 previous similar messages [ 4620.269432] Lustre: Skipped 1 previous similar message [ 4621.284252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4621.286096] LustreError: 312500:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4621.289711] Lustre: Skipped 1 previous similar message [ 4621.297930] LustreError: 312500:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 4623.020083] LustreError: 314348:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4623.022166] LustreError: 314348:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4623.055535] Lustre: server umount lustre-OST0000 complete [ 4626.402453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4626.403421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4626.409823] Lustre: Skipped 1 previous similar message [ 4630.534351] LustreError: 314550:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup [ 4630.537821] LustreError: 314550:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4630.571306] Lustre: server umount lustre-MDT0000 complete [ 4631.520734] LustreError: 310392:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4631.532849] LustreError: 310392:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 4632.198270] LustreError: 311102:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451346 with bad export cookie 12061664876482775061 [ 4632.201769] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4632.202809] LustreError: 311102:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4632.327645] Lustre: server umount lustre-MDT0001 complete [ 4635.791412] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4637.115819] Key type lgssc unregistered [ 4637.259551] LNet: 315284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4637.263735] LNetError: 315284:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4637.273489] LNet: Removed LNI 192.168.203.116@tcp [ 4637.629123] Key type .llcrypt unregistered [ 4637.630663] Key type ._llcrypt unregistered [ 4669.969763] Key type ._llcrypt registered [ 4669.970970] Key type .llcrypt registered [ 4670.004662] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 4676.395722] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4676.855817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4676.911102] alg: No test for adler32 (adler32-zlib) [ 4677.761524] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 4677.842212] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4679.423159] Key type lgssc registered [ 4679.784741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4690.721522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4690.727158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4691.819485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4691.830931] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4691.871959] Lustre: lustre-MDT0000: new disk, initializing [ 4691.898330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4691.904784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4693.305306] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4697.593075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4697.632680] Lustre: 320149:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4697.657890] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4697.659588] Lustre: Skipped 1 previous similar message [ 4697.700045] Lustre: lustre-MDT0001: new disk, initializing [ 4697.731407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4697.744249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4697.748187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4698.953469] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4701.959959] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4703.087773] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4705.351873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4705.468656] Lustre: lustre-OST0000: new disk, initializing [ 4705.470631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4705.499255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4707.360269] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4707.770361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4707.773734] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4707.809886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4710.444165] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4711.830283] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4711.903875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4713.199887] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4713.274179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4718.050502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4718.050920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4718.052056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4718.053220] LustreError: Skipped 1 previous similar message [ 4718.061871] Lustre: Skipped 1 previous similar message [ 4723.169870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4723.171858] Lustre: Skipped 2 previous similar messages [ 4728.287140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4728.290327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4728.291955] Lustre: Skipped 1 previous similar message [ 4728.304121] LustreError: 323075:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4728.346930] Lustre: server umount lustre-OST0000 complete [ 4733.408644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4733.409521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4733.413162] Lustre: Skipped 1 previous similar message [ 4733.416037] Lustre: Skipped 1 previous similar message [ 4735.680149] LustreError: 323279:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4735.682840] LustreError: 323279:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4735.712431] Lustre: server umount lustre-MDT0000 complete [ 4736.947207] LustreError: 323281:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451450 with bad export cookie 6385918544889140069 [ 4736.950548] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4736.951037] LustreError: 323281:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4737.024094] LustreError: 323482:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4737.026186] LustreError: 323482:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4737.075584] Lustre: server umount lustre-MDT0001 complete [ 4739.700931] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:04:12 (1770451452) [ 4741.957396] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:04:15 (1770451455) [ 4744.716154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4744.907724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4746.246316] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4749.033826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4749.156552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4750.476974] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4752.254166] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4753.407910] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4755.626450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4757.615089] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4759.409153] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4773.404144] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4773.922702] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4774.427254] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4774.963219] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4775.583933] Lustre: Failing over lustre-MDT0000 [ 4775.815151] LustreError: 326949:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4775.816931] LustreError: 326949:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4775.841273] Lustre: server umount lustre-MDT0000 complete [ 4779.489636] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4779.490957] LustreError: 324493:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4779.495272] Lustre: Skipped 2 previous similar messages [ 4779.502285] LustreError: 324493:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 4 previous similar messages [ 4780.319251] LustreError: 324488:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4784.608368] LustreError: 325201:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4784.614630] LustreError: 325201:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 4788.681290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4788.719448] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4788.811288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4788.813096] Lustre: Skipped 1 previous similar message [ 4788.833747] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4790.089451] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4790.551266] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4793.830163] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4793.836336] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4793.856545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4794.991183] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4795.516807] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4797.354948] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4797.989225] Lustre: Failing over lustre-MDT0000 [ 4798.107957] LustreError: 328024:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4798.109655] LustreError: 328024:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4798.134260] Lustre: server umount lustre-MDT0000 complete [ 4798.945432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4798.947603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4798.949546] LustreError: 324489:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4798.953946] Lustre: Skipped 1 previous similar message [ 4798.968218] LustreError: 324489:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 4804.064053] LustreError: 326647:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4804.068650] LustreError: 326647:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 4811.570786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4811.616556] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4811.722987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4811.759657] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4813.135762] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4815.157085] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4816.873186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4816.877024] Lustre: Skipped 2 previous similar messages [ 4816.886913] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4816.914845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4818.018845] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4818.537345] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4821.986782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4821.987763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4821.988380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4822.000416] Lustre: Skipped 3 previous similar messages [ 4826.949892] LustreError: 329053:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4826.952162] LustreError: 329053:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4826.984307] Lustre: server umount lustre-MDT0000 complete [ 4827.104105] LustreError: 324492:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4827.108024] LustreError: 324492:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 7 previous similar messages [ 4828.241340] LustreError: 326951:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451542 with bad export cookie 6385918544889142708 [ 4828.244955] LustreError: 326951:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4828.245257] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4828.388078] Lustre: server umount lustre-MDT0001 complete [ 4840.352797] Lustre: server umount lustre-OST0000 complete [ 4845.851847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4846.012793] LustreError: 330283:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4846.035936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4847.308190] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4850.224520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4852.095747] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4854.521630] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4855.913527] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4858.968940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4861.183963] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4863.121742] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4871.397103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4875.744081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4875.746032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4875.749569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4875.752785] Lustre: Skipped 2 previous similar messages [ 4876.973854] LustreError: 332545:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4876.975774] LustreError: 332545:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4877.010576] Lustre: server umount lustre-OST0000 complete [ 4880.285085] Lustre: server umount lustre-MDT0000 complete [ 4881.377167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4881.379422] LustreError: Skipped 1 previous similar message [ 4881.381614] LustreError: 330301:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4881.547591] LustreError: 330263:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451595 with bad export cookie 6385918544889143436 [ 4881.553466] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4881.695604] Lustre: server umount lustre-MDT0001 complete [ 4885.194807] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4886.209948] Key type lgssc unregistered [ 4886.333420] LNet: 333479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4886.335529] LNetError: 333479:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4886.341290] LNet: Removed LNI 192.168.203.116@tcp [ 4886.641600] Key type .llcrypt unregistered [ 4886.643064] Key type ._llcrypt unregistered [ 4896.525500] Key type ._llcrypt registered [ 4896.527214] Key type .llcrypt registered [ 4896.570100] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4897.006532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4897.060992] alg: No test for adler32 (adler32-zlib) [ 4897.930461] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 4898.028485] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4899.615163] Key type lgssc registered [ 4900.017840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4902.516867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4902.522949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4903.700186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4905.051034] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4907.980879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4908.128269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4909.600545] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4911.337528] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4912.598790] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4914.814885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4914.943365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4916.754733] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4918.537312] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4928.299678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 4933.601421] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4933.602309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4933.606582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4933.611514] Lustre: Skipped 2 previous similar messages [ 4938.721984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4938.724021] Lustre: Skipped 1 previous similar message [ 4939.263305] LustreError: 337214:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4939.294533] Lustre: server umount lustre-MDT0000 complete [ 4940.480220] LustreError: 334934:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451654 with bad export cookie 4139151567896290779 [ 4940.483047] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4940.483706] LustreError: 334934:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4940.554045] LustreError: 337418:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4940.557112] LustreError: 337418:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4940.609321] Lustre: server umount lustre-MDT0001 complete [ 4951.029996] LustreError: 337618:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4951.032230] LustreError: 337618:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4951.069854] Lustre: server umount lustre-OST0000 complete [ 4955.378811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4955.545099] LustreError: 338346:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4955.553087] LustreError: 338346:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 4955.569526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4956.885620] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4959.669700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4959.788043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4961.039901] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4963.139183] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4964.447679] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4966.805725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4969.098788] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4971.252895] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4980.261280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4990.435113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4990.436550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4990.437082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4990.440397] Lustre: Skipped 3 previous similar messages [ 4994.497940] LustreError: 340712:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4994.500128] LustreError: 340712:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4994.532206] Lustre: server umount lustre-MDT0000 complete [ 4995.554672] LustreError: 338345:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4995.565365] LustreError: 338345:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 4995.977879] LustreError: 338327:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451709 with bad export cookie 4139151567896291878 [ 4995.982470] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4995.986039] LustreError: 338327:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4996.136544] Lustre: server umount lustre-MDT0001 complete [ 5007.670658] LustreError: 341113:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5007.673752] LustreError: 341113:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5007.719674] Lustre: server umount lustre-OST0000 complete [ 5013.329648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5013.562733] LustreError: 341939:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5013.588963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5013.591023] Lustre: Skipped 1 previous similar message [ 5014.964025] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5017.925386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5019.366557] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5021.928355] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5023.626609] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5026.409909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5028.848454] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5030.888646] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5038.821385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5044.193974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5044.196826] LustreError: Skipped 1 previous similar message [ 5044.196905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5044.199887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5044.206674] Lustre: Skipped 2 previous similar messages [ 5044.210567] Lustre: Skipped 3 previous similar messages [ 5044.846093] LustreError: 344200:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5044.848248] LustreError: 344200:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5044.884590] Lustre: server umount lustre-OST0000 complete [ 5048.300955] Lustre: server umount lustre-MDT0000 complete [ 5048.799761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5048.802208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5048.806393] LustreError: 341939:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5049.656485] LustreError: 341919:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451763 with bad export cookie 4139151567896292886 [ 5049.662884] LustreError: 341919:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5049.663700] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5049.805886] Lustre: server umount lustre-MDT0001 complete [ 5053.331374] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5054.445349] Key type lgssc unregistered [ 5054.591487] LNet: 345133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5054.594136] LNetError: 345133:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5054.606426] LNet: Removed LNI 192.168.203.116@tcp [ 5054.912695] Key type .llcrypt unregistered [ 5054.913720] Key type ._llcrypt unregistered [ 5064.734379] Key type ._llcrypt registered [ 5064.735445] Key type .llcrypt registered [ 5064.770367] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5065.078821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5065.108600] alg: No test for adler32 (adler32-zlib) [ 5065.966342] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 5066.051114] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5067.631168] Key type lgssc registered [ 5068.023062] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5071.367373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5071.373183] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5071.376147] LustreError: 346530:0:(osd_handler.c:8561:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5071.383857] LustreError: 346530:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5071.386044] LustreError: 346530:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5071.388083] LustreError: 346530:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5071.390409] LustreError: 346530:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5074.210035] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5074.214572] LustreError: 346927:0:(osd_handler.c:8561:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5074.225561] LustreError: 346927:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5074.230086] LustreError: 346927:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5074.234517] LustreError: 346927:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5074.239693] LustreError: 346927:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5081.881203] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5083.163087] Key type lgssc unregistered [ 5083.320631] LNet: 347909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5083.325388] LNetError: 347909:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5083.342638] LNet: Removed LNI 192.168.203.116@tcp [ 5083.697473] Key type .llcrypt unregistered [ 5083.699183] Key type ._llcrypt unregistered [ 5090.331508] Key type ._llcrypt registered [ 5090.333057] Key type .llcrypt registered [ 5090.377351] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 5096.263457] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5096.819197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5096.843414] alg: No test for adler32 (adler32-zlib) [ 5097.702446] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 5097.791292] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5099.375136] Key type lgssc registered [ 5099.742458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5110.400947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5110.408097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5111.503602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5111.513634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5111.549033] Lustre: lustre-MDT0000: new disk, initializing [ 5111.572489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5111.578228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5112.885683] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5116.858451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5116.898515] Lustre: 351002:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5116.919837] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5116.921475] Lustre: Skipped 1 previous similar message [ 5116.957224] Lustre: lustre-MDT0001: new disk, initializing [ 5116.986764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5116.998886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5117.001973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5118.289155] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5121.228255] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5122.328178] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5124.520278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5124.636605] Lustre: lustre-OST0000: new disk, initializing [ 5124.639709] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5124.672404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5126.456927] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5126.522746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5126.525874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5126.563697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5129.448505] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5130.784407] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5130.855891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5132.073950] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5132.154747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5136.870275] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5136.870499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5136.872165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5136.874701] LustreError: Skipped 1 previous similar message [ 5136.881542] Lustre: Skipped 1 previous similar message [ 5138.861975] LustreError: 353933:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5138.898090] Lustre: server umount lustre-OST0000 complete [ 5141.986720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5141.988327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5141.994181] Lustre: Skipped 1 previous similar message [ 5141.996712] Lustre: Skipped 1 previous similar message [ 5146.446485] LustreError: 354134:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5146.450683] LustreError: 354134:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5146.492921] Lustre: server umount lustre-MDT0000 complete [ 5147.719141] LustreError: 350993:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451861 with bad export cookie 2819544011384237069 [ 5147.722076] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5147.722858] LustreError: 350993:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5147.784502] LustreError: 354335:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5147.786841] LustreError: 354335:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5147.832832] Lustre: server umount lustre-MDT0001 complete [ 5150.867170] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5151.392313] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:11:04 (1770451864) [ 5154.570950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5154.760064] LustreError: 355052:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5154.768780] LustreError: 355052:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 5154.787605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5156.311191] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5159.077841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5159.195696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5160.509843] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5162.322219] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5163.615925] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5166.540483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5168.897875] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5171.038414] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5173.854615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5173.949371] Lustre: lustre-OST0001: new disk, initializing [ 5173.952086] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5173.990823] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5173.993353] Lustre: Skipped 1 previous similar message [ 5176.476670] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5179.914514] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5179.932634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5179.938897] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5179.995046] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5184.999489] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5184.999863] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5185.002339] LustreError: Skipped 1 previous similar message [ 5185.003173] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5185.006796] Lustre: Skipped 1 previous similar message [ 5188.023234] LustreError: 358245:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5188.026968] LustreError: 358245:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5188.058648] Lustre: server umount lustre-OST0001 complete [ 5190.123170] LustreError: 356648:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5190.133668] LustreError: 356648:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 5195.237690] LustreError: 358362:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5195.243448] LustreError: 358362:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 5200.360316] LustreError: 358362:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5200.365397] LustreError: 358362:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 5200.864295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5200.866664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5200.871096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5200.873859] Lustre: Skipped 2 previous similar messages [ 5204.960046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5204.962636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.967735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5205.484612] LustreError: 358149:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5205.496485] LustreError: 358149:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 5206.766079] LustreError: 358469:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5206.768612] LustreError: 358469:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5206.806359] Lustre: server umount lustre-OST0000 complete [ 5210.593487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5214.629162] Lustre: server umount lustre-MDT0000 complete [ 5215.712243] LustreError: 355763:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5215.718399] LustreError: 355763:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 5215.962169] LustreError: 355034:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451929 with bad export cookie 2819544011384237937 [ 5215.965842] LustreError: 355034:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5215.966937] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5222.079083] LustreError: 358872:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5222.080936] LustreError: 358872:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5222.129612] Lustre: server umount lustre-MDT0001 complete [ 5225.684389] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5226.695191] Key type lgssc unregistered [ 5226.815481] LNet: 359404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5226.817568] LNetError: 359404:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5226.826397] LNet: Removed LNI 192.168.203.116@tcp [ 5227.130901] Key type .llcrypt unregistered [ 5227.131914] Key type ._llcrypt unregistered [ 5243.785077] Key type ._llcrypt registered [ 5243.786046] Key type .llcrypt registered [ 5243.822508] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5244.170053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5244.178137] alg: No test for adler32 (adler32-zlib) [ 5245.032549] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 5245.116953] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5246.695136] Key type lgssc registered [ 5247.035478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5249.492557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5249.498286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5250.590865] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5250.597788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5250.656207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5251.903971] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5254.578157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5254.607829] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5254.618756] Lustre: 362155:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5254.719388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5255.940194] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5257.553838] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5258.619918] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5259.744112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5259.746288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5259.750502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5261.792899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5261.793336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5265.404084] LustreError: 363519:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 5265.425924] Lustre: server umount lustre-MDT0000 complete [ 5266.532203] LustreError: 362880:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451980 with bad export cookie 15918721596153263475 [ 5266.535305] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5266.536861] LustreError: 362880:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5266.598101] LustreError: 363722:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 5266.600255] LustreError: 363722:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5266.647127] Lustre: server umount lustre-MDT0001 complete [ 5269.572420] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:13:02 (1770451982) [ 5271.693164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5271.873307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5273.095885] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5275.756602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5275.870942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5277.069990] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5278.759025] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5279.885474] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5282.004696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5282.081742] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5282.145050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5283.879371] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5285.463555] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5289.076817] Lustre: Permanently deactivating lustre-OST0000 [ 5289.081115] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5292.398747] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5293.453910] LustreError: 367037:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5293.457051] LustreError: 367037:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5293.458923] LustreError: 367037:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5294.027711] LustreError: 367085:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5294.030462] LustreError: 367085:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5300.720153] LustreError: 367182:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5300.722741] LustreError: 367182:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5300.771347] Lustre: server umount lustre-OST0000 complete [ 5302.025801] LustreError: 367335:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5302.027878] LustreError: 367335:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5305.314144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5305.314865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5305.321176] Lustre: Skipped 1 previous similar message [ 5305.324545] Lustre: Skipped 1 previous similar message [ 5310.432721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5310.434466] Lustre: Skipped 2 previous similar messages [ 5312.482853] LustreError: 367433:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5312.484720] LustreError: 367433:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5312.508922] Lustre: server umount lustre-MDT0000 complete [ 5313.681230] LustreError: 366045:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770452027 with bad export cookie 15918721596153264042 [ 5313.684216] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5313.684524] LustreError: 366045:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5323.235304] LustreError: 367634:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5323.237232] LustreError: 367634:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5323.286732] Lustre: server umount lustre-MDT0001 complete [ 5326.019048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5327.176494] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5328.007432] LustreError: 368475:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5330.020479] Lustre: 368813:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.116@tcp [ 5330.700122] Lustre: server umount MGS complete [ 5334.326188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5334.486993] LustreError: 369536:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5334.492078] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5334.492728] LustreError: 369536:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 5334.494217] Lustre: Skipped 1 previous similar message [ 5334.513383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5335.835734] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5338.635114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5338.736957] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5339.981360] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5341.772953] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5342.976156] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5343.586871] Lustre: Permanently reactivating lustre-OST0000 [ 5343.592964] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5343.594871] Lustre: Skipped 2 previous similar messages [ 5369.311146] LustreError: 371094:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5371.812452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5371.958650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5371.960549] Lustre: Skipped 1 previous similar message [ 5374.020847] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5375.870777] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5388.255916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5388.257979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5388.261494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5389.280970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5393.710069] LustreError: 372907:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5393.711748] LustreError: 372907:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5393.742443] Lustre: server umount lustre-OST0000 complete [ 5395.424169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5395.426229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5395.430413] Lustre: Skipped 1 previous similar message [ 5399.520590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5399.522311] Lustre: Skipped 4 previous similar messages [ 5409.247073] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5409.295721] Lustre: server umount lustre-MDT0000 complete [ 5409.761707] LustreError: 369532:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5409.765957] LustreError: 369532:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 5410.438462] LustreError: 369517:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770452124 with bad export cookie 15918721596153265064 [ 5410.443452] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5410.506536] LustreError: 373316:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5410.508839] LustreError: 373316:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5410.567216] Lustre: server umount lustre-MDT0001 complete [ 5413.840169] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5414.877592] Key type lgssc unregistered [ 5415.004350] LNet: 373846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5415.006710] LNetError: 373846:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5415.014360] LNet: Removed LNI 192.168.203.116@tcp [ 5415.294882] Key type .llcrypt unregistered [ 5415.295791] Key type ._llcrypt unregistered [ 5421.533387] Key type ._llcrypt registered [ 5421.534376] Key type .llcrypt registered [ 5421.567241] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 5427.474734] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5427.873175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5427.883644] alg: No test for adler32 (adler32-zlib) [ 5428.725227] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 5428.804719] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5430.383111] Key type lgssc registered [ 5430.700762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5440.934301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5440.939798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5442.038698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5442.049773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5442.091800] Lustre: lustre-MDT0000: new disk, initializing [ 5442.115408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5442.121817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5443.361699] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5447.668255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5447.705659] Lustre: 376940:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5447.726055] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5447.727527] Lustre: Skipped 1 previous similar message [ 5447.766503] Lustre: lustre-MDT0001: new disk, initializing [ 5447.796342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5447.806710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5447.809253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5449.145780] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5452.483819] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5453.594735] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5456.735593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5456.854700] Lustre: lustre-OST0000: new disk, initializing [ 5456.856812] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5456.884287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5458.123531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5458.130373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5458.180766] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5458.905270] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5462.485441] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5463.888411] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5463.988959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5465.366994] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5465.430386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5468.644527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5468.645427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5468.655152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5468.665418] Lustre: Skipped 1 previous similar message [ 5472.236777] LustreError: 379871:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5472.276664] Lustre: server umount lustre-OST0000 complete [ 5473.760870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5473.761073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5473.761799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5473.771401] Lustre: Skipped 2 previous similar messages [ 5478.881151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5478.885575] Lustre: Skipped 2 previous similar messages [ 5479.627050] LustreError: 380072:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5479.630718] LustreError: 380072:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5479.669595] Lustre: server umount lustre-MDT0000 complete [ 5481.360518] LustreError: 376931:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770452195 with bad export cookie 7491089040321469139 [ 5481.367610] LustreError: 376931:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5481.368321] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5481.474283] LustreError: 380274:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5481.478453] LustreError: 380274:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5481.560928] Lustre: server umount lustre-MDT0001 complete [ 5484.489105] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5485.288905] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:16:38 (1770452198) [ 5488.038204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5488.216331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5489.554261] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5493.216916] LustreError: 380992:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5493.223420] LustreError: 380992:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 5493.514841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5493.645273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5495.007348] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5496.740062] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5498.061861] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5500.356357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5502.754614] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5504.524854] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5505.195654] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5516.148347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5519.329068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5519.330383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5519.331090] LustreError: Skipped 1 previous similar message [ 5519.334926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5519.337495] Lustre: Skipped 1 previous similar message [ 5523.756487] LustreError: 383444:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5523.758350] LustreError: 383444:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5523.799454] Lustre: server umount lustre-OST0000 complete [ 5529.568542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5529.568667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5529.575131] Lustre: Skipped 1 previous similar message [ 5529.577208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5529.579142] Lustre: Skipped 1 previous similar message [ 5531.199753] LustreError: 383648:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5531.201465] LustreError: 383648:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5531.224750] Lustre: server umount lustre-MDT0000 complete [ 5532.442743] LustreError: 380971:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770452246 with bad export cookie 7491089040321470007 [ 5532.446921] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5532.449544] LustreError: 380971:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5532.574518] Lustre: server umount lustre-MDT0001 complete [ 5536.618908] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5537.779476] Key type lgssc unregistered [ 5537.904385] LNet: 384380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5537.907135] LNetError: 384380:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5537.913420] LNet: Removed LNI 192.168.203.116@tcp [ 5538.247275] Key type .llcrypt unregistered [ 5538.249073] Key type ._llcrypt unregistered [ 5546.077716] Key type ._llcrypt registered [ 5546.078925] Key type .llcrypt registered [ 5546.123161] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5546.246269] Key type .llcrypt unregistered [ 5546.248413] Key type ._llcrypt unregistered [ 5553.013663] Key type ._llcrypt registered [ 5553.014680] Key type .llcrypt registered [ 5553.053284] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5553.480029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5553.487956] alg: No test for adler32 (adler32-zlib) [ 5554.338427] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 5554.416566] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5555.999144] Key type lgssc registered [ 5556.376843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5559.043598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5559.050188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5560.218492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5561.507760] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5564.794861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5564.912490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5567.049891] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5569.118777] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5570.017477] LustreError: 386527:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5570.020952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5570.022127] LustreError: 386527:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 4 previous similar messages [ 5571.376952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5571.496759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5572.694311] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5580.256105] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5580.258277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5580.261924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5581.796299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5581.797679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5581.797737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5585.903117] LustreError: 388579:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5585.949123] Lustre: server umount lustre-OST0000 complete [ 5589.987116] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5589.991912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5590.000878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5590.005185] Lustre: Skipped 1 previous similar message [ 5592.034318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5592.035346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5593.662045] LustreError: 388780:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5593.663825] LustreError: 388780:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5593.692538] Lustre: server umount lustre-MDT0000 complete [ 5594.907799] LustreError: 388143:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770452308 with bad export cookie 14963436927561489669 [ 5594.913251] LustreError: 388143:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5594.913745] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5594.982023] LustreError: 388982:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5594.985140] LustreError: 388982:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5595.038745] Lustre: server umount lustre-MDT0001 complete [ 5598.563840] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5599.505066] Key type lgssc unregistered [ 5599.625300] LNet: 389513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5599.627527] LNetError: 389513:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5599.633517] LNet: Removed LNI 192.168.203.116@tcp [ 5599.911979] Key type .llcrypt unregistered [ 5599.913063] Key type ._llcrypt unregistered [ 5608.838586] Key type ._llcrypt registered [ 5608.839542] Key type .llcrypt registered [ 5608.902308] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5609.276284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5609.287936] alg: No test for adler32 (adler32-zlib) [ 5610.150362] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 5610.239748] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5611.823144] Key type lgssc registered [ 5612.173808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5614.593380] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5614.598877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5630.047271] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5635.167150] LustreError: 390860:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 5661.791249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5664.149982] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5666.148243] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5668.582175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5668.815276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5668.845509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5670.082636] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5673.803249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5674.013039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5675.679023] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5677.990790] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5679.701538] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5684.193880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5684.193996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5684.200636] LustreError: Skipped 1 previous similar message [ 5684.207437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5684.211780] Lustre: Skipped 1 previous similar message [ 5686.241559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5686.245627] Lustre: Skipped 1 previous similar message [ 5688.431179] LustreError: 393136:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 5688.490503] Lustre: server umount lustre-OST0000 complete [ 5691.361984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5691.363069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5691.368452] Lustre: Skipped 2 previous similar messages [ 5696.399444] LustreError: 393339:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5696.404795] LustreError: 393339:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5696.442996] Lustre: server umount lustre-MDT0000 complete [ 5696.480234] LustreError: 391786:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5696.485083] LustreError: 391786:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 5698.065079] LustreError: 390867:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770452412 with bad export cookie 12437110073914155806 [ 5698.067971] LustreError: 390867:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5698.067990] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5698.136674] LustreError: 393542:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5698.138522] LustreError: 393542:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5698.181477] Lustre: server umount lustre-MDT0001 complete [ 5702.685881] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5703.648681] Key type lgssc unregistered [ 5703.770294] LNet: 394073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5703.772640] LNetError: 394073:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5703.784287] LNet: Removed LNI 192.168.203.116@tcp [ 5704.108608] Key type .llcrypt unregistered [ 5704.110780] Key type ._llcrypt unregistered [ 5715.689050] Key type ._llcrypt registered [ 5715.690134] Key type .llcrypt registered [ 5715.734395] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5716.054193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5716.080062] alg: No test for adler32 (adler32-zlib) [ 5716.946513] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 5717.046183] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5718.623254] Key type lgssc registered [ 5719.015974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5721.706667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5721.712308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5722.874493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5724.144167] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5726.891051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5727.015771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5728.282988] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5729.923694] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5731.003842] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5733.050106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5733.167729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5735.046059] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5736.794256] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5747.494481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5757.726677] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 5757.923238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5757.925105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5757.926383] Lustre: Skipped 2 previous similar messages [ 5759.552777] LustreError: 397708:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5759.581801] Lustre: server umount lustre-MDT0000 complete [ 5760.302444] LustreError: 396165:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.16@tcp arrived at 1770452474 with bad export cookie 2130394468193011017 [ 5760.305590] LustreError: 396165:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5760.899431] LustreError: 396165:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770452474 with bad export cookie 2130394468193010394 [ 5760.904512] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5762.015837] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5762.017989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5762.021441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5762.022842] Lustre: Skipped 3 previous similar messages [ 5763.040275] LustreError: 396158:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5763.040543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5763.044350] LustreError: 396158:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 5765.088142] LustreError: 396158:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5765.088371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5765.093081] LustreError: 396158:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 5766.957407] LustreError: 397911:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5766.959654] LustreError: 397911:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5766.995535] Lustre: server umount lustre-OST0000 complete [ 5769.160363] LustreError: 398266:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5769.163668] LustreError: 398266:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5769.218367] Lustre: server umount lustre-MDT0001 complete [ 5772.688265] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5773.747469] Key type lgssc unregistered [ 5773.881390] LNet: 398796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5773.883718] LNetError: 398796:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5773.889565] LNet: Removed LNI 192.168.203.116@tcp [ 5774.182829] Key type .llcrypt unregistered [ 5774.183854] Key type ._llcrypt unregistered [ 5782.935325] Key type ._llcrypt registered [ 5782.936792] Key type .llcrypt registered [ 5782.987551] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5783.509146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5783.518657] alg: No test for adler32 (adler32-zlib) [ 5784.367098] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 5784.453472] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5786.031174] Key type lgssc registered [ 5786.409386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5788.888025] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5788.893968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5790.046244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5791.208984] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5793.787484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5793.918112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5795.138495] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5796.921344] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5798.396809] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5801.483474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5801.640303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5803.993095] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5805.718940] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5815.974435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5819.871850] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5819.873899] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5819.877491] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5821.213077] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [ 5821.410191] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5823.098987] LustreError: 402429:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5823.118424] Lustre: server umount lustre-MDT0001 complete [ 5826.528504] LustreError: 400879:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5826.528783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5826.529142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5826.529955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5826.529959] Lustre: Skipped 2 previous similar messages [ 5826.532349] LustreError: 400879:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 5830.637743] LustreError: 402633:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5830.639630] LustreError: 402633:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5830.672833] Lustre: server umount lustre-OST0000 complete [ 5831.649196] LustreError: 400879:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5831.845960] LustreError: 402834:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5831.847694] LustreError: 402834:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5831.901270] Lustre: server umount lustre-MDT0000 complete [ 5835.953159] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5836.939124] Key type lgssc unregistered [ 5837.073528] LNet: 403517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5837.075721] LNetError: 403517:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5837.087441] LNet: Removed LNI 192.168.203.116@tcp [ 5837.401516] Key type .llcrypt unregistered [ 5837.402473] Key type ._llcrypt unregistered [ 5845.078210] Key type ._llcrypt registered [ 5845.079259] Key type .llcrypt registered [ 5845.112916] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 5850.885893] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5851.366646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5851.378463] alg: No test for adler32 (adler32-zlib) [ 5852.242797] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 5852.326897] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5853.903223] Key type lgssc registered [ 5854.278839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5870.087652] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5876.629491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5876.635796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5877.740073] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5877.761805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5877.807150] Lustre: lustre-MDT0000: new disk, initializing [ 5877.837048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5877.845407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5879.521376] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5885.320251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5885.412871] Lustre: lustre-OST0000: new disk, initializing [ 5885.416074] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5885.419267] Lustre: Skipped 1 previous similar message [ 5885.442751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5887.098766] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5889.213912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5889.219095] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5889.229925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5890.033716] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5895.105322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5895.142281] Lustre: 407450:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5895.166576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5895.201070] Lustre: lustre-MDT0001: new disk, initializing [ 5895.238068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5895.250247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 5895.253149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 5896.380710] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5899.742474] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5899.805912] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5901.015498] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5901.092676] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5904.158629] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [ 5904.353921] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5904.354560] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5904.357102] Lustre: Skipped 1 previous similar message [ 5908.664197] LustreError: 410856:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5908.683734] Lustre: server umount lustre-MDT0001 complete [ 5914.593068] LustreError: 410183:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5914.596938] LustreError: 410183:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 5 previous similar messages [ 5919.713026] LustreError: 410166:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5919.716578] LustreError: 410166:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 5924.832269] LustreError: 410183:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5924.832605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5924.840974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5924.842596] Lustre: Skipped 2 previous similar messages [ 5926.700089] LustreError: 411127:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 5926.701674] LustreError: 411127:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5926.733981] Lustre: server umount lustre-OST0000 complete [ 5942.239065] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5942.271705] LustreError: 411329:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 5942.273254] LustreError: 411329:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5942.314596] Lustre: server umount lustre-MDT0000 complete [ 5946.366936] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5947.783662] Key type lgssc unregistered [ 5947.929750] LNet: 412014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5947.934402] LNetError: 412014:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5947.948582] LNet: Removed LNI 192.168.203.116@tcp [ 5948.378060] Key type .llcrypt unregistered [ 5948.379083] Key type ._llcrypt unregistered [ 5964.587845] Key type ._llcrypt registered [ 5964.589889] Key type .llcrypt registered [ 5964.649689] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5964.813481] Key type .llcrypt unregistered [ 5964.815901] Key type ._llcrypt unregistered [ 5973.119488] Key type ._llcrypt registered [ 5973.120989] Key type .llcrypt registered [ 5973.173199] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5973.674057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5973.716878] alg: No test for adler32 (adler32-zlib) [ 5974.640936] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 5974.732537] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5976.311178] Key type lgssc registered [ 5976.844267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5988.333734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5988.339991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5989.471370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5989.491990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5989.546666] Lustre: lustre-MDT0000: new disk, initializing [ 5989.587310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5989.599242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5990.998240] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5996.052604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5996.099402] Lustre: 415322:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5996.127300] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5996.129057] Lustre: Skipped 1 previous similar message [ 5996.170588] Lustre: lustre-MDT0001: new disk, initializing [ 5996.216148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5996.232832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5996.237384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5997.787207] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6001.210375] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6003.022913] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6005.856160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6005.980886] Lustre: lustre-OST0000: new disk, initializing [ 6005.982739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6006.011621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6007.941165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6007.952405] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6007.993276] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6008.524298] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6012.342990] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6016.992461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6016.997139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6017.005860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6018.531092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6018.532428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6021.264810] LustreError: 417756:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6021.302764] Lustre: server umount lustre-MDT0000 complete [ 6022.869201] LustreError: 416100:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770452736 with bad export cookie 13514345645325647152 [ 6022.874734] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6022.875211] LustreError: 416100:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6022.949331] LustreError: 417960:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6022.951427] LustreError: 417960:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6023.002892] Lustre: server umount lustre-MDT0001 complete [ 6034.423249] LustreError: 418160:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6034.427473] LustreError: 418160:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6034.501634] Lustre: server umount lustre-OST0000 complete [ 6038.651467] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 03:25:51 (1770452751) [ 6041.881743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6042.160617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6044.137302] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6047.201344] LustreError: 418978:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6047.207671] LustreError: 418978:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 6047.837334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6048.059329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6050.075336] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6052.135746] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6053.819722] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6067.643218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6067.746549] Lustre: Found index 0 for lustre-OST0000, updating log [ 6067.842514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6070.441533] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6072.377357] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6087.333881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6092.773051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6092.774543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6092.776772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6092.776781] Lustre: Skipped 2 previous similar messages [ 6092.794854] Lustre: Skipped 2 previous similar messages [ 6095.537152] LustreError: 421254:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6095.541983] LustreError: 421254:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6095.601983] Lustre: server umount lustre-OST0000 complete [ 6108.618461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6108.619123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6108.622465] Lustre: Skipped 1 previous similar message [ 6108.624334] Lustre: Skipped 3 previous similar messages [ 6113.761420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6113.766018] Lustre: Skipped 2 previous similar messages [ 6114.752282] LustreError: 421509:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6114.753965] LustreError: 421509:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6114.779516] Lustre: server umount lustre-MDT0000 complete [ 6116.051878] LustreError: 418960:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770452830 with bad export cookie 13514345645325648545 [ 6116.057476] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6116.059142] LustreError: 418960:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6116.224854] Lustre: server umount lustre-MDT0001 complete [ 6120.013248] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 03:27:13 (1770452833) [ 6135.519418] LNet: 1 local NIs in recovery (showing 1): 192.168.203.116@tcp [ 6704.487338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6704.709657] LustreError: 425067:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6704.742923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6706.472202] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6709.266532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6709.392870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6710.725917] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6712.747363] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6713.932454] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6727.914274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6728.022713] Lustre: Found index 0 for lustre-OST0000, updating log [ 6728.115945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6730.097244] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6731.179138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6732.662283] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6735.713056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6735.717488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6736.740436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6736.744531] Lustre: Skipped 1 previous similar message [ 6739.760292] LustreError: 427364:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6739.764521] LustreError: 427364:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6739.823480] Lustre: server umount lustre-OST0000 complete [ 6742.625592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6742.627528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6742.633314] Lustre: Skipped 1 previous similar message [ 6742.636468] Lustre: Skipped 1 previous similar message [ 6746.721151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6746.725809] Lustre: Skipped 7 previous similar messages [ 6747.727803] LustreError: 427573:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6747.731700] LustreError: 427573:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6747.747562] LustreError: 425067:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6747.754557] LustreError: 425067:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 6747.773919] Lustre: server umount lustre-MDT0000 complete [ 6748.769390] LustreError: 425067:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6748.784972] LustreError: 425067:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 6749.561161] LustreError: 426734:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770453463 with bad export cookie 13514345645325649630 [ 6749.567919] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6749.569180] LustreError: 426734:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6749.717464] Lustre: server umount lustre-MDT0001 complete [ 6763.231186] LNet: 1 local NIs in recovery (showing 1): 192.168.203.116@tcp [ 6768.895325] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 6773.121229] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6786.796997] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 03:38:19 (1770453499) [ 6791.993483] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6792.836175] Key type lgssc unregistered [ 6792.955408] LNet: 431326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6792.957539] LNetError: 431326:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6858.535345] LNet: Removed LNI 192.168.203.116@tcp [ 6859.039761] Key type .llcrypt unregistered [ 6859.041744] Key type ._llcrypt unregistered [ 6866.771923] Key type ._llcrypt registered [ 6866.773755] Key type .llcrypt registered [ 6866.835964] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6867.299140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6867.310075] alg: No test for adler32 (adler32-zlib) [ 6868.169990] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 6868.245471] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6869.823139] Key type lgssc registered [ 6870.370217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6872.601751] LNet: Added LNI 192.168.203.116@tcp100 [8/256/0/180] [ 6873.029160] LNet: Added LNI 192.168.203.116@tcp101 [8/256/0/180] [ 6873.451801] LNet: Added LNI 192.168.203.116@tcp102 [8/256/0/180] [ 6873.892982] LNet: Added LNI 192.168.203.116@tcp103 [8/256/0/180] [ 6874.313075] LNet: Added LNI 192.168.203.116@tcp104 [8/256/0/180] [ 6874.755153] LNet: Added LNI 192.168.203.116@tcp105 [8/256/0/180] [ 6875.103269] LNet: Added LNI 192.168.203.116@tcp106 [8/256/0/180] [ 6875.468137] LNet: Added LNI 192.168.203.116@tcp107 [8/256/0/180] [ 6875.897070] LNet: Added LNI 192.168.203.116@tcp108 [8/256/0/180] [ 6876.322793] LNet: Added LNI 192.168.203.116@tcp109 [8/256/0/180] [ 6876.745950] LNet: Added LNI 192.168.203.116@tcp110 [8/256/0/180] [ 6877.172847] LNet: Added LNI 192.168.203.116@tcp111 [8/256/0/180] [ 6877.602441] LNet: Added LNI 192.168.203.116@tcp112 [8/256/0/180] [ 6878.020697] LNet: Added LNI 192.168.203.116@tcp113 [8/256/0/180] [ 6878.430056] LNet: Added LNI 192.168.203.116@tcp114 [8/256/0/180] [ 6878.833421] LNet: Added LNI 192.168.203.116@tcp115 [8/256/0/180] [ 6879.283048] LNet: Added LNI 192.168.203.116@tcp116 [8/256/0/180] [ 6879.714802] LNet: Added LNI 192.168.203.116@tcp117 [8/256/0/180] [ 6880.112830] LNet: Added LNI 192.168.203.116@tcp118 [8/256/0/180] [ 6880.529903] LNet: Added LNI 192.168.203.116@tcp119 [8/256/0/180] [ 6880.937711] LNet: Added LNI 192.168.203.116@tcp120 [8/256/0/180] [ 6881.361066] LNet: Added LNI 192.168.203.116@tcp121 [8/256/0/180] [ 6881.800725] LNet: Added LNI 192.168.203.116@tcp122 [8/256/0/180] [ 6882.245258] LNet: Added LNI 192.168.203.116@tcp123 [8/256/0/180] [ 6882.612731] LNet: Added LNI 192.168.203.116@tcp124 [8/256/0/180] [ 6883.010796] LNet: Added LNI 192.168.203.116@tcp125 [8/256/0/180] [ 6883.379263] LNet: Added LNI 192.168.203.116@tcp126 [8/256/0/180] [ 6883.770294] LNet: Added LNI 192.168.203.116@tcp127 [8/256/0/180] [ 6884.211316] LNet: Added LNI 192.168.203.116@tcp128 [8/256/0/180] [ 6884.640877] LNet: Added LNI 192.168.203.116@tcp129 [8/256/0/180] [ 6885.076046] LNet: Added LNI 192.168.203.116@tcp130 [8/256/0/180] [ 6885.475406] LNet: Added LNI 192.168.203.116@tcp131 [8/256/0/180] [ 6885.876621] LNet: Added LNI 192.168.203.116@tcp132 [8/256/0/180] [ 6886.312355] LNet: Added LNI 192.168.203.116@tcp133 [8/256/0/180] [ 6886.680702] LNet: Added LNI 192.168.203.116@tcp134 [8/256/0/180] [ 6887.120731] LNet: Added LNI 192.168.203.116@tcp135 [8/256/0/180] [ 6890.087070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6890.093468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6891.138863] Lustre: 434305:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [ 6891.212400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6891.224035] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6891.260162] Lustre: lustre-MDT0000: new disk, initializing [ 6891.296612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6891.304336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6892.194119] LNetError: 431768:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.116 [ 6892.199101] LNetError: Protocol error connecting to 192.168.203.116@tcp108 at host 192.168.203.116:988: is it running a compatible version of Lustre? [ 6892.786361] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6898.250943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6898.333821] Lustre: 434323:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6898.379576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6898.383567] Lustre: Skipped 1 previous similar message [ 6898.431629] Lustre: lustre-MDT0001: new disk, initializing [ 6898.443280] Lustre: 435106:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [ 6898.447133] Lustre: 435106:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6898.498412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6898.516626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6898.521761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6900.195941] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6900.933783] Lustre: 435431:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [ 6900.940255] Lustre: 435431:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6904.084073] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6905.830525] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6909.012356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6909.190497] Lustre: lustre-OST0000: new disk, initializing [ 6909.193668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6909.237975] Lustre: 436076:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [ 6909.241493] Lustre: 436076:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 6909.260061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6911.510487] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6914.616727] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6916.631074] LustreError: 436900:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6916.660541] Lustre: server umount lustre-OST0000 complete [ 6919.136193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6919.138815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6919.143221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6919.150491] Lustre: 437102:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [ 6919.153040] Lustre: 437102:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6923.840061] LustreError: 437100:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6923.841801] LustreError: 437100:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6923.866313] Lustre: server umount lustre-MDT0000 complete [ 6924.919066] LustreError: 436070:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770453638 with bad export cookie 17722110763579088371 [ 6924.920983] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6924.922207] LustreError: 436070:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6924.981895] LustreError: 437305:0:(obd_class.h:479:obd_check_dev()) Device 7 not setup [ 6924.983768] LustreError: 437305:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6925.029814] Lustre: server umount lustre-MDT0001 complete [ 6927.649231] LNetError: 431767:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.116 [ 6927.652322] LNetError: Protocol error connecting to 192.168.203.116@tcp108 at host 192.168.203.116:988: is it running a compatible version of Lustre? [ 6928.065347] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6928.939301] Key type lgssc unregistered [ 6929.095964] LNet: 437843:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6929.099458] LNetError: 437843:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6929.101406] LNet: Removed LNI 192.168.203.116@tcp [ 6930.143138] LNet: Removed LNI 192.168.203.116@tcp100 [ 6930.144410] LNetError: 437843:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6930.146195] LNetError: 437843:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6930.148103] LNet: Removed LNI 192.168.203.116@tcp101 [ 6931.167122] LNet: Removed LNI 192.168.203.116@tcp102 [ 6931.168454] LNetError: 437843:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6931.171021] LNetError: 437843:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6931.174983] LNet: Removed LNI 192.168.203.116@tcp103 [ 6932.191166] LNet: Removed LNI 192.168.203.116@tcp104 [ 6933.215106] LNet: Removed LNI 192.168.203.116@tcp105 [ 6933.216433] LNetError: 437843:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6933.218222] LNetError: 437843:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 6933.220618] LNet: Removed LNI 192.168.203.116@tcp106 [ 6934.239133] LNet: Removed LNI 192.168.203.116@tcp107 [ 6935.263103] LNet: Removed LNI 192.168.203.116@tcp108 [ 6936.287080] LNet: Removed LNI 192.168.203.116@tcp109 [ 6937.311101] LNet: Removed LNI 192.168.203.116@tcp110 [ 6937.312417] LNetError: 437843:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6937.314226] LNetError: 437843:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 6937.317462] LNet: Removed LNI 192.168.203.116@tcp111 [ 6938.335171] LNet: Removed LNI 192.168.203.116@tcp112 [ 6939.359098] LNet: Removed LNI 192.168.203.116@tcp113 [ 6940.383144] LNet: Removed LNI 192.168.203.116@tcp114 [ 6941.407096] LNet: Removed LNI 192.168.203.116@tcp115 [ 6942.431092] LNet: Removed LNI 192.168.203.116@tcp116 [ 6942.432658] LNet: Removed LNI 192.168.203.116@tcp117 [ 6942.433713] LNet: Removed LNI 192.168.203.116@tcp118 [ 6942.434749] LNet: Removed LNI 192.168.203.116@tcp119 [ 6942.435784] LNet: Removed LNI 192.168.203.116@tcp120 [ 6942.436916] LNet: Removed LNI 192.168.203.116@tcp121 [ 6942.438378] LNet: Removed LNI 192.168.203.116@tcp122 [ 6942.439441] LNet: Removed LNI 192.168.203.116@tcp123 [ 6942.440451] LNet: Removed LNI 192.168.203.116@tcp124 [ 6942.441473] LNet: Removed LNI 192.168.203.116@tcp125 [ 6942.442479] LNet: Removed LNI 192.168.203.116@tcp126 [ 6942.443488] LNet: Removed LNI 192.168.203.116@tcp127 [ 6942.444531] LNet: Removed LNI 192.168.203.116@tcp128 [ 6942.445872] LNet: Removed LNI 192.168.203.116@tcp129 [ 6942.446898] LNet: Removed LNI 192.168.203.116@tcp130 [ 6942.448029] LNet: Removed LNI 192.168.203.116@tcp131 [ 6942.449636] LNet: Removed LNI 192.168.203.116@tcp132 [ 6942.451244] LNet: Removed LNI 192.168.203.116@tcp133 [ 6942.452218] LNet: Removed LNI 192.168.203.116@tcp134 [ 6942.459457] LNet: Removed LNI 192.168.203.116@tcp135 [ 6942.696405] Key type .llcrypt unregistered [ 6942.697417] Key type ._llcrypt unregistered [ 6948.979087] Key type ._llcrypt registered [ 6948.980053] Key type .llcrypt registered [ 6949.010040] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 6954.618383] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6954.891483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6954.899893] alg: No test for adler32 (adler32-zlib) [ 6955.736347] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 6955.808160] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6957.383129] Key type lgssc registered [ 6957.682362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6967.490304] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 03:41:20 (1770453680) [ 6973.810942] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6975.010200] Key type lgssc unregistered [ 6975.147741] LNet: 441844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6975.152146] LNetError: 441844:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6975.161469] LNet: Removed LNI 192.168.203.116@tcp [ 6975.457434] Key type .llcrypt unregistered [ 6975.458632] Key type ._llcrypt unregistered [ 6981.470965] Key type ._llcrypt registered [ 6981.471899] Key type .llcrypt registered [ 6981.505701] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6981.820324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6981.867041] alg: No test for adler32 (adler32-zlib) [ 6982.711388] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 6982.789425] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6984.367093] Key type lgssc registered [ 6984.694611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6987.580581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6987.585507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6988.679909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6988.690612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6988.730542] Lustre: lustre-MDT0000: new disk, initializing [ 6988.753270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6988.759115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6989.966374] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6993.877039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6993.914903] Lustre: 443014:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6993.938924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6993.940535] Lustre: Skipped 1 previous similar message [ 6993.974143] Lustre: lustre-MDT0001: new disk, initializing [ 6994.008416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6994.023058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6994.026380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6995.169804] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6998.035406] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6999.167949] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7001.222411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7001.325638] Lustre: lustre-OST0000: new disk, initializing [ 7001.327452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7001.361324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7002.486985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7002.490104] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7002.533422] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7003.074387] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7005.938545] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7007.050447] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing lnet_if_list [ 7008.377995] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7008.386457] LNet: Added LNI 192.168.203.116@tcp10 [8/256/0/180] [ 7009.889448] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7009.897867] LNet: Added LNI 192.168.203.116@tcp11 [8/256/0/180] [ 7010.951938] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7010.960416] LNet: Added LNI 192.168.203.116@tcp12 [8/256/0/180] [ 7012.078272] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7012.086580] LNet: Added LNI 192.168.203.116@tcp13 [8/256/0/180] [ 7013.209305] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7013.218337] LNet: Added LNI 192.168.203.116@tcp14 [8/256/0/180] [ 7014.445570] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7014.453915] LNet: Added LNI 192.168.203.116@tcp15 [8/256/0/180] [ 7015.598109] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7015.606396] LNet: Added LNI 192.168.203.116@tcp16 [8/256/0/180] [ 7016.788212] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7016.800794] LNet: Added LNI 192.168.203.116@tcp17 [8/256/0/180] [ 7018.722852] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7018.741756] LNet: Added LNI 192.168.203.116@tcp18 [8/256/0/180] [ 7020.621502] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7020.638460] LNet: Added LNI 192.168.203.116@tcp19 [8/256/0/180] [ 7022.663278] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7022.682909] LNet: Added LNI 192.168.203.116@tcp20 [8/256/0/180] [ 7024.598939] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7024.607448] LNet: Added LNI 192.168.203.116@tcp21 [8/256/0/180] [ 7026.630776] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7026.649742] LNet: Added LNI 192.168.203.116@tcp22 [8/256/0/180] [ 7028.648372] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7028.664332] LNet: Added LNI 192.168.203.116@tcp23 [8/256/0/180] [ 7030.321099] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7030.330683] LNet: Added LNI 192.168.203.116@tcp24 [8/256/0/180] [ 7032.463984] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7032.484530] LNet: Added LNI 192.168.203.116@tcp25 [8/256/0/180] [ 7034.675940] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7034.685287] LNet: Added LNI 192.168.203.116@tcp26 [8/256/0/180] [ 7036.532908] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7036.543797] LNet: Added LNI 192.168.203.116@tcp27 [8/256/0/180] [ 7036.553400] LNetError: 442277:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.116 [ 7036.560302] LNetError: Protocol error connecting to 192.168.203.116@tcp26 at host 192.168.203.116:988: is it running a compatible version of Lustre? [ 7038.841497] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7038.856937] LNet: Added LNI 192.168.203.116@tcp28 [8/256/0/180] [ 7038.863348] LNetError: 442274:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.116 [ 7038.867837] LNetError: Protocol error connecting to 192.168.203.116@tcp26 at host 192.168.203.116:988: is it running a compatible version of Lustre? [ 7040.752202] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7040.762181] LNet: Added LNI 192.168.203.116@tcp29 [8/256/0/180] [ 7040.766248] LNetError: 442275:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.116 [ 7040.768891] LNetError: Protocol error connecting to 192.168.203.116@tcp28 at host 192.168.203.116:988: is it running a compatible version of Lustre? [ 7041.184215] LNet: 1 local NIs in recovery (showing 1): 192.168.203.116@tcp28 [ 7042.905998] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7042.926689] LNet: Added LNI 192.168.203.116@tcp30 [8/256/0/180] [ 7043.232614] LNetError: 442276:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.116 [ 7043.237808] LNetError: 442276:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [ 7043.241956] LNetError: Protocol error connecting to 192.168.203.116@tcp28 at host 192.168.203.116:988: is it running a compatible version of Lustre? [ 7043.247390] LNetError: Skipped 1 previous similar message [ 7045.133961] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7045.142895] LNet: Added LNI 192.168.203.116@tcp31 [8/256/0/180] [ 7047.333643] LNetError: 442274:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.116 [ 7047.340288] LNetError: 442274:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 4 previous similar messages [ 7047.345295] LNetError: Protocol error connecting to 192.168.203.116@tcp28 at host 192.168.203.116:988: is it running a compatible version of Lustre? [ 7047.352087] LNetError: Skipped 4 previous similar messages [ 7047.442204] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7047.455298] LNet: Added LNI 192.168.203.116@tcp32 [8/256/0/180] [ 7049.807935] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7049.816836] LNet: Added LNI 192.168.203.116@tcp33 [8/256/0/180] [ 7051.627518] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7051.642627] LNet: Added LNI 192.168.203.116@tcp34 [8/256/0/180] [ 7054.276854] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7054.292222] LNet: Added LNI 192.168.203.116@tcp35 [8/256/0/180] [ 7055.521869] LNetError: 442274:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.116 [ 7055.524650] LNetError: 442274:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 6 previous similar messages [ 7055.526725] LNetError: Protocol error connecting to 192.168.203.116@tcp33 at host 192.168.203.116:988: is it running a compatible version of Lustre? [ 7055.529345] LNetError: Skipped 6 previous similar messages [ 7056.400247] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7056.409874] LNet: Added LNI 192.168.203.116@tcp36 [8/256/0/180] [ 7059.230557] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7059.247058] LNet: Added LNI 192.168.203.116@tcp37 [8/256/0/180] [ 7061.411367] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7061.420424] LNet: Added LNI 192.168.203.116@tcp38 [8/256/0/180] [ 7064.365925] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7064.376359] LNet: Added LNI 192.168.203.116@tcp39 [8/256/0/180] [ 7066.580991] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7066.591466] LNet: Added LNI 192.168.203.116@tcp40 [8/256/0/180] [ 7069.462945] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7069.475426] LNet: Added LNI 192.168.203.116@tcp41 [8/256/0/180] [ 7071.910925] LNetError: 442277:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.116 [ 7071.913900] LNetError: 442277:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 27 previous similar messages [ 7071.916761] LNetError: Protocol error connecting to 192.168.203.116@tcp34 at host 192.168.203.116:988: is it running a compatible version of Lustre? [ 7071.919729] LNetError: Skipped 28 previous similar messages [ 7071.999883] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7072.010542] LNet: Added LNI 192.168.203.116@tcp42 [8/256/0/180] [ 7074.955920] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7074.970050] LNet: Added LNI 192.168.203.116@tcp43 [8/256/0/180] [ 7077.045403] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7077.064144] LNet: Added LNI 192.168.203.116@tcp44 [8/256/0/180] [ 7080.066840] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7080.085366] LNet: Added LNI 192.168.203.116@tcp45 [8/256/0/180] [ 7082.214324] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7082.229590] LNet: Added LNI 192.168.203.116@tcp46 [8/256/0/180] [ 7085.546867] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7085.555561] LNet: Added LNI 192.168.203.116@tcp47 [8/256/0/180] [ 7087.653698] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7087.665053] LNet: Added LNI 192.168.203.116@tcp48 [8/256/0/180] [ 7090.340643] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7090.352333] LNet: Added LNI 192.168.203.116@tcp49 [8/256/0/180] [ 7092.877655] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7092.896619] LNet: Added LNI 192.168.203.116@tcp50 [8/256/0/180] [ 7101.599119] LNet: 8 local NIs in recovery (showing 8): 192.168.203.116@tcp44, 192.168.203.116@tcp47, 192.168.203.116@tcp46, 192.168.203.116@tcp42, 192.168.203.116@tcp34, 192.168.203.116@tcp33, 192.168.203.116@tcp32, 192.168.203.116@tcp29 [ 7104.995679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7104.996737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7105.006332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7105.014225] Lustre: Skipped 1 previous similar message [ 7105.698890] LNetError: 442275:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.116 [ 7105.705228] LNetError: 442275:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 40 previous similar messages [ 7105.710328] LNetError: Protocol error connecting to 192.168.203.116@tcp46 at host 192.168.203.116:988: is it running a compatible version of Lustre? [ 7105.717035] LNetError: Skipped 38 previous similar messages [ 7107.630101] LustreError: 467196:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7107.664357] Lustre: server umount lustre-OST0000 complete [ 7110.114175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7110.115656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7110.122277] Lustre: Skipped 2 previous similar messages [ 7110.130622] Lustre: Skipped 1 previous similar message [ 7115.024120] LustreError: 467397:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7115.028105] LustreError: 467397:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7115.067287] Lustre: server umount lustre-MDT0000 complete [ 7116.579036] LustreError: 443004:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770453830 with bad export cookie 17893699186243710960 [ 7116.581965] LustreError: 443004:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7116.582453] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7116.643227] LustreError: 467599:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7116.644962] LustreError: 467599:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7116.686115] Lustre: server umount lustre-MDT0001 complete [ 7120.857445] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7122.340185] Key type lgssc unregistered [ 7122.527302] LNet: 468138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7122.530345] LNetError: 468138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7122.532860] LNet: Removed LNI 192.168.203.116@tcp [ 7123.551097] LNet: Removed LNI 192.168.203.116@tcp10 [ 7123.552343] LNetError: 468138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7123.554223] LNetError: 468138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7123.556283] LNet: Removed LNI 192.168.203.116@tcp11 [ 7124.575220] LNet: Removed LNI 192.168.203.116@tcp12 [ 7124.577797] LNetError: 468138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7124.582161] LNetError: 468138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7124.586411] LNet: Removed LNI 192.168.203.116@tcp13 [ 7125.599103] LNet: Removed LNI 192.168.203.116@tcp14 [ 7126.623211] LNet: Removed LNI 192.168.203.116@tcp15 [ 7126.626077] LNetError: 468138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7126.630903] LNetError: 468138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7126.636029] LNet: Removed LNI 192.168.203.116@tcp16 [ 7127.647189] LNet: Removed LNI 192.168.203.116@tcp17 [ 7128.671163] LNet: Removed LNI 192.168.203.116@tcp18 [ 7129.695202] LNet: Removed LNI 192.168.203.116@tcp19 [ 7130.719160] LNet: Removed LNI 192.168.203.116@tcp20 [ 7130.721491] LNetError: 468138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7130.725097] LNetError: 468138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7130.729229] LNet: Removed LNI 192.168.203.116@tcp21 [ 7131.743179] LNet: Removed LNI 192.168.203.116@tcp22 [ 7132.767218] LNet: Removed LNI 192.168.203.116@tcp23 [ 7133.791164] LNet: Removed LNI 192.168.203.116@tcp24 [ 7134.815108] LNet: Removed LNI 192.168.203.116@tcp25 [ 7135.839108] LNet: Removed LNI 192.168.203.116@tcp26 [ 7136.863183] LNet: Removed LNI 192.168.203.116@tcp27 [ 7137.887179] LNet: Removed LNI 192.168.203.116@tcp28 [ 7137.890181] LNet: Removed LNI 192.168.203.116@tcp29 [ 7138.911222] LNet: Removed LNI 192.168.203.116@tcp30 [ 7138.914166] LNetError: 468138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7138.919125] LNetError: 468138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 7138.924271] LNet: Removed LNI 192.168.203.116@tcp31 [ 7138.926983] LNet: Removed LNI 192.168.203.116@tcp32 [ 7139.935171] LNet: Removed LNI 192.168.203.116@tcp33 [ 7139.938186] LNet: Removed LNI 192.168.203.116@tcp34 [ 7140.959170] LNet: Removed LNI 192.168.203.116@tcp35 [ 7141.983172] LNet: Removed LNI 192.168.203.116@tcp36 [ 7143.007090] LNet: Removed LNI 192.168.203.116@tcp37 [ 7144.031167] LNet: Removed LNI 192.168.203.116@tcp38 [ 7145.055187] LNet: Removed LNI 192.168.203.116@tcp39 [ 7146.079167] LNet: Removed LNI 192.168.203.116@tcp40 [ 7147.103170] LNet: Removed LNI 192.168.203.116@tcp41 [ 7147.106044] LNet: Removed LNI 192.168.203.116@tcp42 [ 7148.127164] LNet: Removed LNI 192.168.203.116@tcp43 [ 7148.129962] LNet: Removed LNI 192.168.203.116@tcp44 [ 7149.151181] LNet: Removed LNI 192.168.203.116@tcp45 [ 7149.153989] LNet: Removed LNI 192.168.203.116@tcp46 [ 7149.156578] LNet: Removed LNI 192.168.203.116@tcp47 [ 7149.159119] LNet: Removed LNI 192.168.203.116@tcp48 [ 7149.161684] LNet: Removed LNI 192.168.203.116@tcp49 [ 7149.170704] LNet: Removed LNI 192.168.203.116@tcp50 [ 7149.527601] LNetError: 468201:0:(module.c:692:libcfs_exit()) Portals memory leaked: 1128 bytes [ 7149.554256] Key type .llcrypt unregistered [ 7149.556180] Key type ._llcrypt unregistered [ 7157.417502] Key type ._llcrypt registered [ 7157.418607] Key type .llcrypt registered [ 7157.451526] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7157.810328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7157.837876] alg: No test for adler32 (adler32-zlib) [ 7158.679160] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 7158.752146] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7160.327195] Key type lgssc registered [ 7160.678894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7163.362137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7163.368328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7164.537996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7165.764953] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7169.003159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7169.147585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7170.591771] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7172.957410] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7174.664438] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7177.590737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7177.737942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7179.990815] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7182.312278] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7197.229271] loop0: detected capacity change from 0 to 204800000 [ 7197.300562] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7197.304491] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7197.308585] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7199.995759] loop0: detected capacity change from 0 to 204800000 [ 7200.104118] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7200.108596] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7200.114102] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7202.189051] loop0: detected capacity change from 0 to 204800000 [ 7204.419746] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7204.498969] Lustre: fs15246-OST0000: new disk, initializing [ 7204.503448] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7204.530226] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7206.681059] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7211.174460] loop1: detected capacity change from 0 to 204800000 [ 7213.314723] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7213.363430] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7213.391279] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7213.432856] Lustre: fs15246-MDT0000: new disk, initializing [ 7213.489662] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7213.500724] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7213.543043] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7214.560498] Lustre: 473746:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000e47060bc x1856455439319168/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1770453944 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7214.895997] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7223.647160] LustreError: 473746:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7223.649917] LustreError: 473746:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 7223.652465] LustreError: 473746:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 7227.363612] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7227.367286] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7228.730262] LustreError: 474867:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 7228.750469] Lustre: server umount fs15246-MDT0000 complete [ 7231.410275] LustreError: 475369:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 7231.411954] LustreError: 475369:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7231.425479] Lustre: server umount fs15246-OST0000 complete [ 7235.039586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7235.041609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7235.045087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7236.064448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7236.066359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7236.069736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7240.428096] LustreError: 475869:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7240.429779] LustreError: 475869:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7240.459801] Lustre: server umount lustre-OST0000 complete [ 7242.720593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7242.721217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7242.723918] Lustre: Skipped 1 previous similar message [ 7242.725401] Lustre: Skipped 2 previous similar messages [ 7247.675474] LustreError: 476071:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7247.677235] LustreError: 476071:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7247.701670] Lustre: server umount lustre-MDT0000 complete [ 7247.840475] LustreError: 469363:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7247.846633] LustreError: 469363:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 7248.789946] LustreError: 474868:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770453962 with bad export cookie 12403464126727215539 [ 7248.792975] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7248.793042] LustreError: 474868:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7248.897617] Lustre: server umount lustre-MDT0001 complete [ 7252.090441] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7252.922716] Key type lgssc unregistered [ 7253.035298] LNet: 476803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7253.037268] LNetError: 476803:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7253.047282] LNet: Removed LNI 192.168.203.116@tcp [ 7253.295364] Key type .llcrypt unregistered [ 7253.296397] Key type ._llcrypt unregistered [ 7259.878297] Key type ._llcrypt registered [ 7259.879338] Key type .llcrypt registered [ 7259.909417] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7260.272021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7260.277389] alg: No test for adler32 (adler32-zlib) [ 7261.113281] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 7261.186971] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7262.759083] Key type lgssc registered [ 7263.052885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7274.005366] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 7277.057630] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7289.261912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7289.266351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7290.344487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7290.354211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7290.388868] Lustre: lustre-MDT0000: new disk, initializing [ 7290.410067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7290.415686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7291.495387] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7295.176030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7295.214715] Lustre: 481182:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7295.233623] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7295.235250] Lustre: Skipped 1 previous similar message [ 7295.269462] Lustre: lustre-MDT0001: new disk, initializing [ 7295.297347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7295.307497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7295.311656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7296.369751] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7299.155177] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7300.203908] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7302.114771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7302.203559] Lustre: lustre-OST0000: new disk, initializing [ 7302.205309] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7302.228775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7303.791975] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7306.559564] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7307.681137] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7308.275757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7308.278417] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7308.306453] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7308.752934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 7309.825593] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7309.882426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7313.376754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7313.378394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7313.378627] LustreError: Skipped 1 previous similar message [ 7313.381967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7313.383931] Lustre: Skipped 1 previous similar message [ 7316.468055] LustreError: 484118:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7316.496181] Lustre: server umount lustre-OST0000 complete [ 7318.496389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7318.497038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7318.499346] Lustre: Skipped 2 previous similar messages [ 7323.576221] LustreError: 484320:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7323.577796] LustreError: 484320:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7323.605870] Lustre: server umount lustre-MDT0000 complete [ 7324.703122] LustreError: 481173:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770454038 with bad export cookie 9291313350241581742 [ 7324.706027] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7324.708516] LustreError: 481173:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7324.770303] LustreError: 484523:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7324.771967] LustreError: 484523:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7324.823721] Lustre: server umount lustre-MDT0001 complete [ 7332.267520] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 7335.194640] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7344.364896] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7161 sec ========== 03:47:37 (1770454057) [ 7344.865123] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:47:37 (1770454057) === [ 7345.927635] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:47:39 (1770454059) === [ 7358.257837] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7359.062495] Key type lgssc unregistered [ 7359.169329] LNet: 489359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7359.171464] LNetError: 489359:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7359.180270] LNet: Removed LNI 192.168.203.116@tcp [ 7359.423846] Key type .llcrypt unregistered [ 7359.424846] Key type ._llcrypt unregistered