[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 513150725 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002398] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.006862] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007033] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008018] pid_max: default: 32768 minimum: 301 [ 0.009203] LSM: Security Framework initializing [ 0.010066] Yama: becoming mindful. [ 0.011056] SELinux: Initializing. [ 0.012092] *** VALIDATE selinux *** [ 0.020247] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024887] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026091] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027135] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028169] *** VALIDATE tmpfs *** [ 0.030466] *** VALIDATE proc *** [ 0.031318] *** VALIDATE cgroup *** [ 0.032016] *** VALIDATE cgroup2 *** [ 0.033354] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035198] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036017] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037049] Spectre V2 : User space: Vulnerable [ 0.038011] Speculative Store Bypass: Vulnerable [ 0.041611] debug: unmapping init [mem 0xffffffffb3259000-0xffffffffb3260fff] [ 0.043277] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044801] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045040] ... version: 2 [ 0.046025] ... bit width: 48 [ 0.047017] ... generic registers: 4 [ 0.048018] ... value mask: 0000ffffffffffff [ 0.049020] ... max period: 00007fffffffffff [ 0.050022] ... fixed-purpose events: 3 [ 0.051021] ... event mask: 000000070000000f [ 0.052480] rcu: Hierarchical SRCU implementation. [ 0.055301] smp: Bringing up secondary CPUs ... [ 0.056762] x86: Booting SMP configuration: [ 0.057066] .... node #0, CPUs: #1 #2 #3 [ 0.061119] smp: Brought up 1 node, 4 CPUs [ 0.063023] smpboot: Max logical packages: 1 [ 0.064021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.288900] node 0 deferred pages initialised in 222ms [ 0.292178] devtmpfs: initialized [ 0.293359] x86/mm: Memory block size: 128MB [ 0.296974] gcov: version magic: 0x41383552 [ 0.300322] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.303167] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.305418] pinctrl core: initialized pinctrl subsystem [ 0.307203] [ 0.307673] ************************************************************* [ 0.310013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.312014] ** ** [ 0.314015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.316014] ** ** [ 0.318019] ** This means that this kernel is built to expose internal ** [ 0.322016] ** IOMMU data structures, which may compromise security on ** [ 0.324015] ** your system. ** [ 0.327025] ** ** [ 0.329014] ** If you see this message and you are not debugging the ** [ 0.332017] ** kernel, report this immediately to your vendor! ** [ 0.335020] ** ** [ 0.337018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.339016] ************************************************************* [ 0.341765] NET: Registered protocol family 16 [ 0.343477] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.346075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.348076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.352037] cpuidle: using governor menu [ 0.353785] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.357757] PCI: Using configuration type 1 for base access [ 0.360160] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.369152] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.372080] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.375112] cryptd: max_cpu_qlen set to 1000 [ 0.377270] ACPI: Added _OSI(Module Device) [ 0.379028] ACPI: Added _OSI(Processor Device) [ 0.381016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.382028] ACPI: Added _OSI(Processor Aggregator Device) [ 0.387514] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.393401] ACPI: Interpreter enabled [ 0.394048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.396011] ACPI: Using IOAPIC for interrupt routing [ 0.397118] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.401416] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.410609] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.413047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.416041] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.419103] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.423502] acpiphp: Slot [2] registered [ 0.425127] acpiphp: Slot [5] registered [ 0.427206] acpiphp: Slot [6] registered [ 0.428166] acpiphp: Slot [7] registered [ 0.430223] acpiphp: Slot [8] registered [ 0.431132] acpiphp: Slot [9] registered [ 0.432148] acpiphp: Slot [10] registered [ 0.434160] acpiphp: Slot [3] registered [ 0.435133] acpiphp: Slot [4] registered [ 0.436107] acpiphp: Slot [11] registered [ 0.437106] acpiphp: Slot [12] registered [ 0.438081] acpiphp: Slot [13] registered [ 0.439128] acpiphp: Slot [14] registered [ 0.440110] acpiphp: Slot [15] registered [ 0.441077] acpiphp: Slot [16] registered [ 0.442082] acpiphp: Slot [17] registered [ 0.443077] acpiphp: Slot [18] registered [ 0.444044] acpiphp: Slot [19] registered [ 0.445113] acpiphp: Slot [20] registered [ 0.446148] acpiphp: Slot [21] registered [ 0.448101] acpiphp: Slot [22] registered [ 0.449028] acpiphp: Slot [23] registered [ 0.450090] acpiphp: Slot [24] registered [ 0.450923] acpiphp: Slot [25] registered [ 0.451063] acpiphp: Slot [26] registered [ 0.452083] acpiphp: Slot [27] registered [ 0.453149] acpiphp: Slot [28] registered [ 0.454059] acpiphp: Slot [29] registered [ 0.454944] acpiphp: Slot [30] registered [ 0.456107] acpiphp: Slot [31] registered [ 0.457046] PCI host bridge to bus 0000:00 [ 0.458019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.460029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.462034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.464024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.466033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.469053] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.470221] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.471934] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.475313] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.483644] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.489100] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.492035] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.493020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.495018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.498033] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.500883] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.503038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.504752] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.510022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.524029] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.529027] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.535623] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.544021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.551032] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.568031] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.576974] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.601025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.616027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.638027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.650431] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.661025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.670019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.681023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.689352] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.695018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.715019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.748029] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.776716] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.782023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.788015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.802023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.810159] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.815021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.819019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.829021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.839347] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.842419] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.844612] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.847434] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.850362] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.856217] iommu: Default domain type: Passthrough [ 0.858605] SCSI subsystem initialized [ 0.860149] ACPI: bus type USB registered [ 0.861121] usbcore: registered new interface driver usbfs [ 0.863103] usbcore: registered new interface driver hub [ 0.866136] usbcore: registered new device driver usb [ 0.868184] pps_core: LinuxPPS API ver. 1 registered [ 0.870024] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.874088] PTP clock support registered [ 0.877061] EDAC MC: Ver: 3.0.0 [ 0.878654] PCI: Using ACPI for IRQ routing [ 0.881458] NetLabel: Initializing [ 0.883015] NetLabel: domain hash size = 128 [ 0.885017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.887119] NetLabel: unlabeled traffic allowed by default [ 0.890057] vgaarb: loaded [ 0.891385] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.894023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.903169] clocksource: Switched to clocksource kvm-clock [ 1.013445] VFS: Disk quotas dquot_6.6.0 [ 1.015476] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.019137] *** VALIDATE ramfs *** [ 1.020444] *** VALIDATE hugetlbfs *** [ 1.022406] pnp: PnP ACPI init [ 1.025788] pnp: PnP ACPI: found 6 devices [ 1.050741] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.054313] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.056680] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.058817] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.061353] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.063924] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.066175] NET: Registered protocol family 2 [ 1.068618] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.073622] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.077722] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.083549] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.086954] TCP: Hash tables configured (established 65536 bind 65536) [ 1.090116] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.093393] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.097130] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.100362] NET: Registered protocol family 1 [ 1.103456] RPC: Registered named UNIX socket transport module. [ 1.105669] RPC: Registered udp transport module. [ 1.107402] RPC: Registered tcp transport module. [ 1.109320] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.112417] NET: Registered protocol family 44 [ 1.114051] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.116330] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.118618] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.121264] PCI: CLS 0 bytes, default 64 [ 1.123615] Unpacking initramfs... [ 2.587636] debug: unmapping init [mem 0xffff92613cc54000-0xffff92613ffbffff] [ 2.592246] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.594642] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.598114] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.173841] Initialise system trusted keyrings [ 3.175624] Key type blacklist registered [ 3.177740] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.187333] zbud: loaded [ 3.190596] *** VALIDATE nfs *** [ 3.191966] *** VALIDATE nfs4 *** [ 3.193731] pstore: using deflate compression [ 3.197282] Platform Keyring initialized [ 3.306895] NET: Registered protocol family 38 [ 3.309472] Key type asymmetric registered [ 3.311077] Asymmetric key parser 'x509' registered [ 3.313082] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.316724] io scheduler mq-deadline registered [ 3.319071] io scheduler kyber registered [ 3.321048] io scheduler bfq registered [ 3.323281] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.327362] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.331750] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.336096] ACPI: Power Button [PWRF] [ 3.342174] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.349592] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.362607] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.369792] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.384598] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.411444] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.441626] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.447404] Non-volatile memory driver v1.3 [ 3.449388] Linux agpgart interface v0.103 [ 3.485632] virtio_blk virtio1: [vda] 134216 512-byte logical blocks (68.7 MB/65.5 MiB) [ 3.488306] vda: detected capacity change from 0 to 68718592 [ 3.505417] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.508319] vdb: detected capacity change from 0 to 1073741824 [ 3.528746] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.531135] vdc: detected capacity change from 0 to 2621440000 [ 3.545743] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.548257] vdd: detected capacity change from 0 to 2621440000 [ 3.561398] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.563946] vde: detected capacity change from 0 to 4294967296 [ 3.577665] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.580029] vdf: detected capacity change from 0 to 4294967296 [ 3.585680] libphy: Fixed MDIO Bus: probed [ 3.601643] usbcore: registered new interface driver usbserial_generic [ 3.604621] usbserial: USB Serial support registered for generic [ 3.607722] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.612362] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.614382] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.617279] mousedev: PS/2 mouse device common for all mice [ 3.621054] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.622353] rtc_cmos 00:05: RTC can wake from S4 [ 3.634896] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.637093] rtc_cmos 00:05: registered as rtc0 [ 3.646834] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.648600] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.648647] intel_pstate: CPU model not supported [ 3.653902] hid: raw HID events driver (C) Jiri Kosina [ 3.661201] usbcore: registered new interface driver usbhid [ 3.663366] usbhid: USB HID core driver [ 3.665222] drop_monitor: Initializing network drop monitor service [ 3.667959] Initializing XFRM netlink socket [ 3.670030] NET: Registered protocol family 10 [ 3.673428] Segment Routing with IPv6 [ 3.674948] NET: Registered protocol family 17 [ 3.676824] mpls_gso: MPLS GSO support [ 3.682690] RAS: Correctable Errors collector initialized. [ 3.684880] AVX version of gcm_enc/dec engaged. [ 3.686522] AES CTR mode by8 optimization enabled [ 3.762842] sched_clock: Marking stable (3762821657, 0)->(4804692003, -1041870346) [ 3.766620] registered taskstats version 1 [ 3.768320] Loading compiled-in X.509 certificates [ 3.770739] zswap: loaded using pool lzo/zbud [ 3.797370] Key type big_key registered [ 3.811054] Key type encrypted registered [ 3.812359] ima: No TPM chip found, activating TPM-bypass! [ 3.814610] ima: Allocated hash algorithm: sha1 [ 3.817327] ima: No architecture policies found [ 3.819402] evm: Initialising EVM extended attributes: [ 3.821227] evm: security.selinux [ 3.822341] evm: security.ima [ 3.823435] evm: security.capability [ 3.824744] evm: HMAC attrs: 0x1 [ 3.827093] rtc_cmos 00:05: setting system clock to 2026-01-30 06:02:35 UTC (1769752955) [ 3.833211] debug: unmapping init [mem 0xffffffffb4203000-0xffffffffb43fffff] [ 3.836290] debug: unmapping init [mem 0xffffffffb2f82000-0xffffffffb3258fff] [ 3.840435] Write protecting the kernel read-only data: 28672k [ 3.843314] debug: unmapping init [mem 0xffffffffb1603000-0xffffffffb17fffff] [ 3.846344] debug: unmapping init [mem 0xffffffffb1f14000-0xffffffffb1ffffff] [ 3.882221] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.890696] systemd[1]: Detected virtualization kvm. [ 3.893068] systemd[1]: Detected architecture x86-64. [ 3.895161] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.920631] systemd[1]: No hostname configured. [ 3.922494] systemd[1]: Set hostname to . [ 3.924944] random: systemd: uninitialized urandom read (16 bytes read) [ 3.927707] systemd[1]: Initializing machine ID from random generator. [ 4.074077] random: systemd: uninitialized urandom read (16 bytes read) [ 4.076898] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.081510] random: systemd: uninitialized urandom read (16 bytes read) [ 4.084490] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.090057] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.781347] device-mapper: uevent: version 1.0.3 [ 4.783526] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System...[ 5.389224] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.641655] virtio_net virtio0 ens2: renamed from eth0 [ 5.700975] scsi host0: ata_piix [ 5.743988] scsi host1: ata_piix [ 5.745404] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.747849] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.274065] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.333203] random: crng init done [ 10.334323] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.898229] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.160957] printk: systemd: 22 output lines suppressed due to ratelimiting [ 12.413354] SELinux: Disabled at runtime. [ 12.477791] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.488263] systemd[1]: Detected virtualization kvm. [ 12.490315] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.961586] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.964585] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.969111] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.971996] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.975251] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.982570] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.986146] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System.[ 13.184754] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.489108] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.794397] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.823094] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.005077] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.018739] EDAC sbridge: Ver: 1.1.2 [ 15.674706] Key type dns_resolver registered [ 15.971089] NFS: Registering the id_resolver key type [ 15.973138] Key type id_resolver registered [ 15.974877] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg423-server login: [ 39.609873] hrtimer: interrupt took 9887494 ns [ 67.716978] libcfs: loading out-of-tree module taints kernel. [ 67.783566] Key type ._llcrypt registered [ 67.786530] Key type .llcrypt registered [ 67.857697] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 88.542777] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 90.251216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 90.263408] alg: No test for adler32 (adler32-zlib) [ 91.767090] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 92.622292] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 94.347644] Key type lgssc registered [ 96.216249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 110.709845] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 162.389588] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 179.213215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 179.265874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.725771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 180.772180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 180.918650] Lustre: lustre-MDT0000: new disk, initializing [ 181.074490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.123639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 185.522054] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.732054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 205.936902] Lustre: 6506:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 206.073903] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 206.089761] Lustre: Skipped 1 previous similar message [ 206.280447] Lustre: lustre-MDT0001: new disk, initializing [ 206.379593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 206.440356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 206.455846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 212.050989] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.747727] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 229.599316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 229.868429] Lustre: lustre-OST0000: new disk, initializing [ 229.878365] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 229.954762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 231.429978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 231.448948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 231.559457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 237.109493] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 254.326197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 254.449057] Lustre: lustre-OST0001: new disk, initializing [ 254.454793] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 254.514405] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 260.152904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 260.161299] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 260.281308] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 261.503887] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 276.363350] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 285.095211] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 297.036920] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing check_logdir /tmp/testlogs/ [ 302.427974] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing yml_node [ 309.617945] Lustre: DEBUG MARKER: Client: 2.17.50.72 [ 312.827926] Lustre: DEBUG MARKER: MDS: 2.17.50.72 [ 316.838969] Lustre: DEBUG MARKER: OSS: 2.17.50.72 [ 320.172405] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jan 30 01:07:49 EST 2026 [ 343.091931] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 345.245104] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 362.481368] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.489469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 362.497182] Lustre: Skipped 3 previous similar messages [ 362.515438] Lustre: Skipped 3 previous similar messages [ 367.585431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 367.591783] Lustre: Skipped 2 previous similar messages [ 368.679059] LustreError: 12211:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 368.978809] Lustre: server umount lustre-MDT0000 complete [ 373.347895] LustreError: 6498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769753325 with bad export cookie 5563379761740018687 [ 373.354004] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 373.384803] LustreError: 6498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 377.829149] LustreError: 10442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 377.829906] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 377.854294] LustreError: 10442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 377.879955] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 377.886156] Lustre: Skipped 2 previous similar messages [ 378.849430] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 378.884134] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 379.594311] LustreError: 12413:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 379.608506] LustreError: 12413:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 380.128079] Lustre: server umount lustre-MDT0001 complete [ 385.223149] LustreError: 12616:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 385.245625] LustreError: 12616:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 385.351749] Lustre: server umount lustre-OST0000 complete [ 389.339250] LustreError: 12816:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 389.343973] LustreError: 12816:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 389.494312] Lustre: server umount lustre-OST0001 complete [ 397.820047] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 409.524514] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 451.596584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 452.033634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 452.101513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 452.255893] Lustre: lustre-MDT0000: new disk, initializing [ 452.351279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 452.379826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 457.143786] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 467.558432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 467.702291] Lustre: 14989:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 467.787277] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 467.790059] Lustre: Skipped 1 previous similar message [ 467.854138] Lustre: lustre-MDT0001: new disk, initializing [ 467.927247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 467.981032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 467.998056] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 472.502500] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 480.556283] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 485.442889] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 493.123466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 493.423909] Lustre: lustre-OST0000: new disk, initializing [ 493.430435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 493.522900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.942307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 494.949237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 495.083795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 500.278520] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 508.462615] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 513.021946] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 513.203732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 517.775657] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 518.036345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 520.678731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 520.682354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 520.702935] LustreError: Skipped 1 previous similar message [ 520.704166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 520.704172] Lustre: Skipped 2 previous similar messages [ 520.711297] Lustre: Skipped 2 previous similar messages [ 525.805042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 525.816565] Lustre: Skipped 2 previous similar messages [ 526.089986] LustreError: 17920:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 526.114075] LustreError: 17920:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 526.346773] Lustre: server umount lustre-OST0000 complete [ 530.914357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.925386] Lustre: Skipped 1 previous similar message [ 536.032699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 536.040427] Lustre: Skipped 3 previous similar messages [ 545.247204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 545.409223] LustreError: 18122:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 545.423725] LustreError: 18122:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 545.671778] Lustre: server umount lustre-MDT0000 complete [ 546.281704] LustreError: 15002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.298433] LustreError: 15002:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 550.378197] LustreError: 15002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.417528] LustreError: 15002:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 552.318452] LustreError: 14982:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769753503 with bad export cookie 5563379761740020220 [ 552.330249] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 552.338479] LustreError: 14982:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 552.983943] Lustre: server umount lustre-MDT0001 complete [ 559.947488] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:11:49 (1769753509) [ 562.877310] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 565.147908] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:11:55 (1769753515) [ 573.266820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.931259] LustreError: 18995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.042767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 578.928454] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 584.172173] LustreError: 18995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.199907] LustreError: 18995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 588.811401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 589.281554] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 589.297268] Lustre: Skipped 2 previous similar messages [ 594.172930] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 600.242497] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 604.971586] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 612.542451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 619.532160] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 626.340791] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 635.367037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 635.375711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.386342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 639.458365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 640.799761] LustreError: 21235:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 640.808833] LustreError: 21235:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 641.138650] Lustre: server umount lustre-OST0000 complete [ 645.600830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 645.619092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 645.632625] Lustre: Skipped 1 previous similar message [ 651.788706] Lustre: server umount lustre-MDT0000 complete [ 654.816398] LustreError: 18991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 654.834251] LustreError: 18991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 656.898401] LustreError: 21441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769753608 with bad export cookie 5563379761740021102 [ 656.914196] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.921529] LustreError: 21441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 657.407881] Lustre: server umount lustre-MDT0001 complete [ 671.596474] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 674.826512] Key type lgssc unregistered [ 675.247096] LNet: 22175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.267276] LNetError: 22175:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 675.319822] LNet: Removed LNI 192.168.204.123@tcp [ 676.203126] Key type .llcrypt unregistered [ 676.205489] Key type ._llcrypt unregistered [ 702.837322] Key type ._llcrypt registered [ 702.839680] Key type .llcrypt registered [ 703.051047] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 704.714745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 704.794747] alg: No test for adler32 (adler32-zlib) [ 706.289803] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 706.765622] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 708.607307] Key type lgssc registered [ 710.023772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 720.247563] loop: module loaded [ 720.354348] loop0: detected capacity change from 0 to 204800000 [ 720.571671] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 720.601850] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 720.904738] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 726.864510] loop0: detected capacity change from 0 to 204800000 [ 727.126406] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 727.143592] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 727.166622] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 733.179726] loop0: detected capacity change from 0 to 204800000 [ 738.839455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 738.868568] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 740.040162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 740.085276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 740.144775] Lustre: lustre-MDT0000: new disk, initializing [ 740.228801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 740.247404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 744.878264] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 754.141558] loop1: detected capacity change from 0 to 204800000 [ 759.915511] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 760.375151] Lustre: lustre-OST07c6: new disk, initializing [ 760.378750] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 760.394844] Lustre: Skipped 1 previous similar message [ 760.470587] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 766.089784] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 768.632077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 768.642977] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 768.688480] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 772.125255] Lustre: 24333:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 772.146594] Lustre: 24333:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 772.158460] Lustre: 24333:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 772.179676] Lustre: 24333:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 772.190327] Lustre: 24333:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 772.202571] Lustre: 24333:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 786.247808] Lustre: 24331:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 786.259265] Lustre: 24331:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 786.270553] Lustre: 24331:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 786.282996] Lustre: 24331:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 786.290846] Lustre: 24331:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 786.311017] Lustre: 24331:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 791.263295] Lustre: 24331:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 791.276187] Lustre: 24331:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 791.288696] Lustre: 24331:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 791.306572] Lustre: 24331:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 791.318784] Lustre: 24331:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 791.336595] Lustre: 24331:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 791.346607] Lustre: 24331:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 791.360782] Lustre: 24331:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 791.379575] Lustre: 24331:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 791.390725] Lustre: 24331:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 791.404613] Lustre: 24331:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 791.427827] Lustre: 24331:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.015803] Lustre: 24332:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 797.025836] Lustre: 24332:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 797.033065] Lustre: 24332:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 797.038452] Lustre: 24332:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.043855] Lustre: 24332:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 797.049620] Lustre: 24332:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.054113] Lustre: 24332:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 797.060130] Lustre: 24332:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.065310] Lustre: 24332:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 797.079229] Lustre: 24332:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.091939] Lustre: 24332:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 797.103612] Lustre: 24332:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 801.900926] Lustre: 24331:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 801.914207] Lustre: 24331:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 801.916887] Lustre: 24331:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 801.923305] Lustre: 24331:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 801.930607] Lustre: 24331:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 801.939966] Lustre: 24331:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 801.947424] Lustre: 24331:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 801.954544] Lustre: 24331:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 801.959901] Lustre: 24331:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 801.966747] Lustre: 24331:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 801.975192] Lustre: 24331:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 801.981730] Lustre: 24331:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.284757] Lustre: 25784:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 814.296412] Lustre: 25784:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 5 previous similar messages [ 814.305064] Lustre: 25784:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 814.315389] Lustre: 25784:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 814.322898] Lustre: 25784:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 814.344498] Lustre: 25784:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 814.354071] Lustre: 25784:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 814.364259] Lustre: 25784:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 814.369694] Lustre: 25784:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 814.373799] Lustre: 25784:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 814.382388] Lustre: 25784:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 814.391747] Lustre: 25784:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 865.763726] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 865.772088] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 865.793139] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 867.636183] LustreError: 26512:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 867.836817] Lustre: server umount lustre-OST07c6 complete [ 876.723229] LustreError: 27013:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 876.730131] LustreError: 27013:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 876.999432] Lustre: server umount lustre-MDT0000 complete [ 895.694317] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 899.014926] Key type lgssc unregistered [ 899.387961] LNet: 27852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.393184] LNetError: 27852:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.412038] LNet: Removed LNI 192.168.204.123@tcp [ 900.449168] Key type .llcrypt unregistered [ 900.452546] Key type ._llcrypt unregistered [ 921.333214] Key type ._llcrypt registered [ 921.337606] Key type .llcrypt registered [ 921.439732] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 922.450996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 922.593305] alg: No test for adler32 (adler32-zlib) [ 923.759813] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 924.003454] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 925.743180] Key type lgssc registered [ 927.040943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 934.349205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 934.367791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 935.806393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 939.861498] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 946.146034] LustreError: 29110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 946.178859] LustreError: 29110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 948.019885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 948.255611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 953.139901] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 958.576576] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 963.434519] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 971.551608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 972.083494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 978.318311] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 981.303529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 985.787715] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1001.958467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1001.959906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1001.961328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1001.980522] Lustre: Skipped 1 previous similar message [ 1007.077939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1007.083767] Lustre: Skipped 2 previous similar messages [ 1007.546158] LustreError: 31557:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1007.851444] Lustre: server umount lustre-OST0000 complete [ 1012.195156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1012.198530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1012.208955] Lustre: Skipped 1 previous similar message [ 1012.220993] Lustre: Skipped 1 previous similar message [ 1017.318037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1017.326852] Lustre: Skipped 1 previous similar message [ 1017.991331] LustreError: 31760:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1018.006559] LustreError: 31760:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1018.257464] Lustre: server umount lustre-MDT0000 complete [ 1022.374421] LustreError: 29091:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769753974 with bad export cookie 16546561202017283305 [ 1022.381508] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1022.387502] LustreError: 29091:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1022.541374] LustreError: 31962:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1022.549822] LustreError: 31962:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1023.014663] Lustre: server umount lustre-MDT0001 complete [ 1037.779744] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1041.224518] Key type lgssc unregistered [ 1041.519408] LNet: 32494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.529313] LNetError: 32494:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.559723] LNet: Removed LNI 192.168.204.123@tcp [ 1042.492259] Key type .llcrypt unregistered [ 1042.494828] Key type ._llcrypt unregistered [ 1067.205757] Key type ._llcrypt registered [ 1067.208227] Key type .llcrypt registered [ 1067.306417] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1069.162309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1069.213170] alg: No test for adler32 (adler32-zlib) [ 1070.319774] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 1070.583599] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1072.231173] Key type lgssc registered [ 1073.833863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1084.522913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1084.552575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1086.149721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.575972] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1095.651511] LustreError: 33745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1095.694644] LustreError: 33745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1100.459719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1100.770778] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1101.068323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1106.112553] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1112.416702] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1117.271138] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1126.010204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1126.581415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1130.722764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1134.088759] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1142.476719] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1161.699596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1161.701909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.720970] LustreError: Skipped 1 previous similar message [ 1161.723148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1161.778097] Lustre: Skipped 1 previous similar message [ 1165.127780] LustreError: 36011:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1165.441676] Lustre: server umount lustre-OST0000 complete [ 1171.942750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1171.948248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1171.960819] Lustre: Skipped 1 previous similar message [ 1171.972263] Lustre: Skipped 1 previous similar message [ 1176.308791] LustreError: 36212:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1176.314076] LustreError: 36212:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1176.592122] Lustre: server umount lustre-MDT0000 complete [ 1177.057301] LustreError: 34953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1177.076361] LustreError: 34953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1180.625715] LustreError: 33726:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769754132 with bad export cookie 15460734696321392880 [ 1180.633799] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1180.636437] LustreError: 33726:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1180.741096] LustreError: 36414:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1180.745197] LustreError: 36414:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1180.900890] Lustre: server umount lustre-MDT0001 complete [ 1193.793842] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1196.802300] Key type lgssc unregistered [ 1197.222732] LNet: 36946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1197.229676] LNetError: 36946:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1197.276223] LNet: Removed LNI 192.168.204.123@tcp [ 1198.260173] Key type .llcrypt unregistered [ 1198.267581] Key type ._llcrypt unregistered [ 1223.237587] Key type ._llcrypt registered [ 1223.239811] Key type .llcrypt registered [ 1223.358594] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1224.683554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1224.859929] alg: No test for adler32 (adler32-zlib) [ 1225.974945] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 1226.246334] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1227.951249] Key type lgssc registered [ 1229.093201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1236.235815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1236.263913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.760913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1242.212251] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1248.235208] LustreError: 38347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.248374] LustreError: 38347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1252.318802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1252.919757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1258.676831] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1265.208197] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1269.523944] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1276.870275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1277.220583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.362870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1283.120694] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1289.432331] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1297.664659] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 1297.891650] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.906682] Lustre: Skipped 1 previous similar message [ 1302.761229] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 1302.769462] Lustre: Skipped 3 previous similar messages [ 1303.462428] LustreError: 40588:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1305.524625] LustreError: 40588:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 1305.539252] LustreError: 40588:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1305.568609] Lustre: server umount lustre-MDT0000 complete [ 1307.879566] LustreError: 39247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1309.468380] LustreError: 38327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769754261 with bad export cookie 17539238991437339364 [ 1309.475743] LustreError: 38327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1309.477661] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1313.004644] LustreError: 39247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1313.030155] LustreError: 39247:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1313.033922] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.23@tcp (stopping) [ 1313.056504] Lustre: Skipped 3 previous similar messages [ 1313.251974] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.272885] Lustre: Skipped 1 previous similar message [ 1315.796555] LustreError: 40791:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1315.800735] LustreError: 40791:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1316.039169] Lustre: server umount lustre-MDT0001 complete [ 1332.140663] LustreError: 41017:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1332.149965] LustreError: 41017:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1332.376245] Lustre: server umount lustre-OST0000 complete [ 1349.366925] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1352.379049] Key type lgssc unregistered [ 1352.742945] LNet: 41855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1352.754784] LNetError: 41855:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1353.778475] LNet: Removed LNI 192.168.204.123@tcp [ 1354.599188] Key type .llcrypt unregistered [ 1354.604140] Key type ._llcrypt unregistered [ 1380.635264] Key type ._llcrypt registered [ 1380.636906] Key type .llcrypt registered [ 1380.797584] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1382.067811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1382.227298] alg: No test for adler32 (adler32-zlib) [ 1383.259914] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 1383.498099] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1385.175180] Key type lgssc registered [ 1386.673792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1396.115478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1396.144126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1397.542833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1401.631773] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1407.976772] LustreError: 43252:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1408.003092] LustreError: 43252:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1409.003126] LustreError: 43252:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1411.675452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1412.073167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1417.490820] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1423.476254] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1428.468558] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1436.633130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1436.937655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1444.177450] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1444.180507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1451.617225] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1461.730293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1461.732954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1461.745806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1461.756479] Lustre: Skipped 1 previous similar message [ 1464.583471] Lustre: lustre-OST0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 1466.856338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1466.862311] Lustre: Skipped 1 previous similar message [ 1467.782832] LustreError: 45494:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1468.049217] Lustre: server umount lustre-OST0000 complete [ 1488.712462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1488.726077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1488.736506] Lustre: Skipped 2 previous similar messages [ 1488.755890] Lustre: Skipped 1 previous similar message [ 1494.984032] LustreError: 45871:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1494.994172] LustreError: 45871:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1495.205964] Lustre: server umount lustre-MDT0000 complete [ 1497.066805] LustreError: 43248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1497.122765] LustreError: 43248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1500.654762] LustreError: 43232:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769754452 with bad export cookie 17451001266084145733 [ 1500.676137] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1506.922757] LustreError: 46074:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1506.934594] LustreError: 46074:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1507.270554] Lustre: server umount lustre-MDT0001 complete [ 1522.294497] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1524.724533] Key type lgssc unregistered [ 1525.099618] LNet: 46607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1525.106813] LNetError: 46607:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.183834] LNet: Removed LNI 192.168.204.123@tcp [ 1527.392109] Key type .llcrypt unregistered [ 1527.395162] Key type ._llcrypt unregistered [ 1557.838262] Key type ._llcrypt registered [ 1557.841476] Key type .llcrypt registered [ 1557.941454] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1559.620327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1559.687129] alg: No test for adler32 (adler32-zlib) [ 1560.910293] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 1561.274327] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1563.071970] Key type lgssc registered [ 1564.237945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1572.200775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1572.221356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1573.802153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1579.074845] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1584.100179] LustreError: 48005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1584.132340] LustreError: 48005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1588.419332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1588.867103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1593.449187] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1599.518593] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1604.019167] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1614.713084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1615.246313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1618.355821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1623.729845] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1631.788828] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1640.092133] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1645.663728] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1659.087651] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-01-30 1h30m07s [ 1661.484494] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1664.483136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1664.490810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1664.503036] Lustre: Skipped 2 previous similar messages [ 1664.525183] Lustre: Skipped 2 previous similar messages [ 1665.584847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1665.591730] Lustre: Skipped 3 previous similar messages [ 1670.452446] LustreError: 50573:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1670.692765] Lustre: server umount lustre-MDT0000 complete [ 1671.653931] LustreError: 48005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1671.695149] LustreError: 48005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1674.919506] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1677.795493] LustreError: 49750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1677.821106] LustreError: 49750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1682.161901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1682.317568] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1682.637931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1682.698225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1683.693245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1687.099575] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1691.190652] Lustre: DEBUG MARKER: Wait for df (36112) ... [ 1692.972738] Lustre: DEBUG MARKER: done [ 1698.276577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1698.286841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.313369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1698.324633] Lustre: Skipped 1 previous similar message [ 1699.298682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1704.265699] LustreError: 51633:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1704.276412] LustreError: 51633:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1704.402951] Lustre: server umount lustre-OST0000 complete [ 1709.537062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1709.544533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1709.549486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1709.549498] Lustre: Skipped 2 previous similar messages [ 1716.714206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1716.720693] Lustre: Skipped 4 previous similar messages [ 1722.336522] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1722.499357] LustreError: 51834:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1722.510091] LustreError: 51834:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1722.691646] Lustre: server umount lustre-MDT0000 complete [ 1725.866960] LustreError: 48006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1725.887287] LustreError: 48006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1726.851570] LustreError: 50302:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769754678 with bad export cookie 14058298718999748399 [ 1726.855761] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1726.873315] LustreError: 50302:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1727.067080] LustreError: 52037:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1727.073873] LustreError: 52037:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1727.311581] Lustre: server umount lustre-MDT0001 complete [ 1737.731969] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1740.592679] Key type lgssc unregistered [ 1740.865115] LNet: 52569:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1740.871296] LNetError: 52569:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1740.887683] LNet: Removed LNI 192.168.204.123@tcp [ 1741.622846] Key type .llcrypt unregistered [ 1741.630933] Key type ._llcrypt unregistered [ 1783.421751] Key type ._llcrypt registered [ 1783.423861] Key type .llcrypt registered [ 1783.594182] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1785.025275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1785.239914] alg: No test for adler32 (adler32-zlib) [ 1786.364967] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 1786.597836] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1788.287210] Key type lgssc registered [ 1789.278386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1797.119472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1797.131925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.425776] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1798.441908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1798.602923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1803.120542] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1813.140917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1813.212958] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1813.235203] Lustre: 54757:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1813.397963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1818.366765] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1825.079456] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1829.142138] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1836.017332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1836.190222] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1836.357421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1842.885162] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1844.870813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1848.826217] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1856.583116] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-01-30 1h33m26s [ 1858.552754] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1866.204916] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1869.323021] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1869.370421] LustreError: 54765:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1889.608636] Lustre: lustre-MDT0000: Client be58cfdf-d453-431b-bbcb-ede9c782c067 (at 192.168.204.23@tcp) reconnecting [ 1909.375458] LustreError: 54765:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1912.188479] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1915.003683] Lustre: DEBUG MARKER: done [ 1922.019372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.022217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1922.038476] Lustre: Skipped 1 previous similar message [ 1922.060784] Lustre: Skipped 1 previous similar message [ 1927.145468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1927.150013] Lustre: Skipped 1 previous similar message [ 1932.262983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1932.269390] Lustre: Skipped 1 previous similar message [ 1936.352079] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1936.437477] LustreError: 57794:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1936.526981] Lustre: server umount lustre-OST0000 complete [ 1941.474555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1941.494154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1941.511810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1949.181113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1949.186537] Lustre: Skipped 4 previous similar messages [ 1955.295217] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1955.455766] LustreError: 57997:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1955.467173] LustreError: 57997:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1955.849410] Lustre: server umount lustre-MDT0000 complete [ 1960.217088] LustreError: 54750:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769754911 with bad export cookie 15144604277439246066 [ 1960.234461] LustreError: 54750:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1960.237726] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1960.378246] LustreError: 58200:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1960.394440] LustreError: 58200:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1960.628983] Lustre: server umount lustre-MDT0001 complete [ 1972.967236] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1975.878661] Key type lgssc unregistered [ 1976.319986] LNet: 58734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1976.333526] LNetError: 58734:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1976.377300] LNet: Removed LNI 192.168.204.123@tcp [ 1977.506631] Key type .llcrypt unregistered [ 1977.513846] Key type ._llcrypt unregistered [ 2026.080790] Key type ._llcrypt registered [ 2026.083231] Key type .llcrypt registered [ 2026.177570] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2027.254793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2027.322096] alg: No test for adler32 (adler32-zlib) [ 2028.436883] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 2028.725771] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2030.391154] Key type lgssc registered [ 2031.504493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2034.898699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2034.914671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2036.157774] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2036.187085] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2036.369571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2037.271126] Lustre: Failing over lustre-MDT0000 [ 2037.433877] LustreError: 60958:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2037.589229] Lustre: server umount lustre-MDT0000 complete [ 2044.260162] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:36:34 (1769754994) [ 2051.059840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.545772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.714833] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2064.159215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2064.233799] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2064.254822] Lustre: 61515:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2064.509070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2068.505594] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2074.096469] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2078.131810] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2085.721919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2085.994448] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2086.182190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.950631] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2094.182840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2097.637688] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2103.363485] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2109.921327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2109.927901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2109.952298] Lustre: Skipped 2 previous similar messages [ 2109.958545] Lustre: Skipped 1 previous similar message [ 2115.050608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2115.059445] Lustre: Skipped 3 previous similar messages [ 2120.162362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2120.181373] Lustre: Skipped 2 previous similar messages [ 2124.255215] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2124.353458] LustreError: 63881:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2124.362698] LustreError: 63881:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2124.465278] Lustre: server umount lustre-MDT0000 complete [ 2128.388398] LustreError: 61507:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755080 with bad export cookie 10134961289122296199 [ 2128.390499] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2128.397652] LustreError: 61507:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2128.543840] LustreError: 64085:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2128.551223] LustreError: 64085:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2128.675182] Lustre: server umount lustre-MDT0001 complete [ 2132.755147] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2138.223363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2146.788120] Lustre: 60439:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769755082/real 1769755082] req@000000009d77b65b x1855722597948928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769755098 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2146.819344] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.603392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2149.855199] Lustre: 60437:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769755085/real 1769755085] req@000000006c1c2841 x1855722597949056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769755101 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2151.903625] Lustre: 60436:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769755087/real 1769755087] req@00000000f582dc5d x1855722597949184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769755103 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2154.016248] LustreError: 60435:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006c1c2841 x1855722597949824/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2154.388958] LustreError: 63137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.399289] LustreError: 63137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2154.512367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2154.637991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2155.424952] Lustre: 60439:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769755090/real 1769755090] req@0000000074f46927 x1855722597949440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769755106 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2155.562569] LustreError: 63137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2155.564539] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2156.641944] LustreError: 64942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2156.653687] LustreError: 64942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2158.941879] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2159.695130] Lustre: 60439:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769755095/real 1769755095] req@00000000e11ba1ed x1855722597949568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769755111 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2162.146649] LustreError: 63136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.163031] LustreError: 63136:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2167.273167] LustreError: 64943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2167.295632] LustreError: 64943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2167.993903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2168.728468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2173.938382] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2175.250114] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2182.451180] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2187.177670] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2199.522377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.524067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2199.536724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2199.565743] Lustre: Skipped 1 previous similar message [ 2204.644087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2204.655248] Lustre: Skipped 4 previous similar messages [ 2210.273795] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2210.409799] LustreError: 66346:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2210.418202] LustreError: 66346:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2210.518450] Lustre: server umount lustre-MDT0000 complete [ 2214.267502] LustreError: 61507:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755165 with bad export cookie 10134961289122298481 [ 2214.270266] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2214.281708] LustreError: 61507:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2214.509220] LustreError: 66550:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2214.516188] LustreError: 66550:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2214.668327] Lustre: server umount lustre-MDT0001 complete [ 2221.563697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2231.265225] Lustre: 60439:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769755166/real 1769755166] req@0000000003d9f63c x1855722597978496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769755182 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2231.291697] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2231.310996] Lustre: Skipped 1 previous similar message [ 2232.690829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2238.495819] LustreError: 60435:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009926ed09 x1855722597979264/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2238.887174] LustreError: 63136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.987982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2239.038918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2240.051895] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2245.416637] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2255.659852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2261.497138] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2261.544586] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2268.260630] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2273.044276] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2279.930416] LustreError: 68807:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2279.938521] LustreError: 68807:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2280.034537] Lustre: server umount lustre-MDT0000 complete [ 2281.956231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2281.962815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2281.968288] LustreError: 67407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.971847] Lustre: Skipped 1 previous similar message [ 2282.022321] LustreError: 67407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 2284.287268] LustreError: 61506:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755235 with bad export cookie 10134961289122299860 [ 2284.293521] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2284.301794] LustreError: 61506:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2284.810810] Lustre: server umount lustre-MDT0001 complete [ 2291.664951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2295.665592] Lustre: DEBUG MARKER: files compared the same [ 2306.575983] LustreError: 69604:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2306.582642] LustreError: 69604:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2306.741328] Lustre: server umount lustre-OST0000 complete [ 2325.416295] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2328.153953] Key type lgssc unregistered [ 2328.437032] LNet: 70443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.442507] LNetError: 70443:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2328.460279] LNet: Removed LNI 192.168.204.123@tcp [ 2329.387790] Key type .llcrypt unregistered [ 2329.393213] Key type ._llcrypt unregistered [ 2353.796666] Key type ._llcrypt registered [ 2353.798726] Key type .llcrypt registered [ 2353.897874] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2354.934906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2354.987257] alg: No test for adler32 (adler32-zlib) [ 2355.948867] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 2356.169199] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2357.823217] Key type lgssc registered [ 2358.781193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2366.358272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2366.380348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2367.922334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2371.965323] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug +malloc all [ 2378.210800] LustreError: 71838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2378.224127] LustreError: 71838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2380.935960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2381.396176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2385.293576] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug +malloc all [ 2392.056656] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2396.029795] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2403.664832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2403.938035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2410.606309] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug +malloc all [ 2413.164655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2416.722411] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2427.880078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2427.887379] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2427.896666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2428.897391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2428.900695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2428.908306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.929424] Lustre: Skipped 1 previous similar message [ 2431.553881] LustreError: 74080:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2431.728501] Lustre: server umount lustre-OST0000 complete [ 2437.600058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2437.622134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2437.650629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2441.402600] LustreError: 74283:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2441.407860] LustreError: 74283:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2441.632990] Lustre: server umount lustre-MDT0000 complete [ 2444.260412] LustreError: 71838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.281356] LustreError: 71838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2445.948119] LustreError: 71820:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755397 with bad export cookie 6584099056659018466 [ 2445.949863] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2445.956652] LustreError: 71820:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2446.159489] LustreError: 74485:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2446.172671] LustreError: 74485:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2446.323826] Lustre: server umount lustre-MDT0001 complete [ 2457.946359] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2460.863256] Key type lgssc unregistered [ 2461.181613] LNet: 75017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2461.203509] LNetError: 75017:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2461.221849] LNet: Removed LNI 192.168.204.123@tcp [ 2461.923951] Key type .llcrypt unregistered [ 2461.925437] Key type ._llcrypt unregistered [ 2488.865727] Key type ._llcrypt registered [ 2488.869245] Key type .llcrypt registered [ 2489.050635] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2490.394804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2490.447171] alg: No test for adler32 (adler32-zlib) [ 2491.498264] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 2491.746744] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2493.439174] Key type lgssc registered [ 2494.520680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2503.128745] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2503.142191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2518.687421] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2523.807929] LustreError: 76393:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.123@tcp: failed processing log, type 4: rc = -110 [ 2550.495211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2558.411320] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2564.904873] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2572.472642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2572.603242] Lustre: *** cfs_fail_loc=706, val=0*** [ 2572.626583] LustreError: 75730:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000058a702e5 x1855723083401472/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2573.094781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2573.166594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2577.347658] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2578.408571] LustreError: 76418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.442136] LustreError: 76418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2583.537924] LustreError: 76418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.562843] LustreError: 77368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2586.249721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2586.662480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2590.716513] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2596.585243] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2600.822557] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2604.009084] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2604.021386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.052746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2607.074542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2607.083132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2607.098069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.345497] LustreError: 78719:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2609.412681] Lustre: server umount lustre-OST0000 complete [ 2617.313891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2617.316211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2617.328840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2617.343762] Lustre: Skipped 2 previous similar messages [ 2619.974335] LustreError: 78921:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2619.994389] LustreError: 78921:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2620.324056] Lustre: server umount lustre-MDT0000 complete [ 2622.433721] LustreError: 77371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2622.449640] LustreError: 77371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2624.080105] LustreError: 76399:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755575 with bad export cookie 16630397025061137617 [ 2624.092966] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2624.262729] LustreError: 79122:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2624.279139] LustreError: 79122:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2624.716097] Lustre: server umount lustre-MDT0001 complete [ 2636.407387] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2638.980794] Key type lgssc unregistered [ 2639.206416] LNet: 79655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2639.218185] LNetError: 79655:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2639.231686] LNet: Removed LNI 192.168.204.123@tcp [ 2639.973406] Key type .llcrypt unregistered [ 2639.975223] Key type ._llcrypt unregistered [ 2668.598573] Key type ._llcrypt registered [ 2668.600791] Key type .llcrypt registered [ 2668.691354] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2669.774587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2669.864407] alg: No test for adler32 (adler32-zlib) [ 2670.872692] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 2671.212882] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2672.960053] Key type lgssc registered [ 2673.985039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2682.078790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2682.095525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.910604] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2696.706226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2697.425624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2702.062848] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2702.819383] LustreError: 81722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.853628] LustreError: 81722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2707.937330] LustreError: 81727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2707.952801] LustreError: 81727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2712.274991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2712.745828] LustreError: 81722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2712.773819] LustreError: 81722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2712.797846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2718.183747] LustreError: 81726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2718.212431] LustreError: 81726:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2720.294407] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2723.328323] LustreError: 81726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.367319] LustreError: 81726:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2731.078331] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2731.094759] Lustre: MGS: Client 98131e2f-2749-4b71-b8fe-29ba0b174ea6 (at 0@lo) reconnecting [ 2731.125331] LustreError: 81737:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2731.493879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2731.652368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2737.099523] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2751.972049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2751.988123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2752.001838] Lustre: Skipped 1 previous similar message [ 2752.031404] Lustre: Skipped 1 previous similar message [ 2757.105094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2757.109488] Lustre: Skipped 1 previous similar message [ 2757.328682] LustreError: 83684:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2757.479918] Lustre: server umount lustre-OST0000 complete [ 2762.217090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2762.218338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.249064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2767.329220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2767.331915] Lustre: Skipped 1 previous similar message [ 2768.147347] LustreError: 83887:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2768.156800] LustreError: 83887:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2768.282944] Lustre: server umount lustre-MDT0000 complete [ 2772.454111] LustreError: 81723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2772.478561] LustreError: 81723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2772.681291] LustreError: 84089:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2772.688683] LustreError: 84089:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2773.279994] Lustre: server umount lustre-MDT0001 complete [ 2778.141039] LustreError: 84291:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2778.154540] LustreError: 84291:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2778.227701] Lustre: server umount MGS complete [ 2792.321837] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:49:02 (1769755742) [ 2825.425184] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 2837.762948] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2878.077978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2883.579889] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2896.406158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2896.660348] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2896.707290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2896.787582] Lustre: lustre-MDT0001: new disk, initializing [ 2896.896451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2896.961120] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2896.971075] Lustre: 88800:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2896.979399] LustreError: 88800:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2902.365751] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2916.173862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2916.515285] Lustre: lustre-OST0000: new disk, initializing [ 2916.523982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2916.641841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2924.582195] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2927.145458] Lustre: 89959:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000dd40fb09 x1855723271136640/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1769755894 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2933.096620] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2940.974069] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2941.000042] Lustre: MGS: Client 51196a59-e6e5-4a51-bdc8-cedff3e01803 (at 0@lo) reconnecting [ 2941.052813] Lustre: 88083:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2941.091075] LustreError: 88786:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2941.101578] LustreError: 88786:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2941.132716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2941.214977] Lustre: lustre-MDT0000: new disk, initializing [ 2941.456248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2941.511560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2941.548748] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2942.568572] Lustre: 90510:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000377de45e x1855723271143296/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1769755910 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2946.480751] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2947.807947] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2951.676799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2951.696858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2951.876312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2977.762705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2977.769845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2977.778079] Lustre: Skipped 1 previous similar message [ 2977.789482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2980.335280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2980.356926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2982.429859] LustreError: 91031:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2982.434890] LustreError: 91031:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2982.770501] Lustre: server umount lustre-OST0000 complete [ 2991.076763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2991.078841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2991.090374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2991.090383] Lustre: Skipped 2 previous similar messages [ 2991.131788] Lustre: Skipped 1 previous similar message [ 2993.937636] LustreError: 91233:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2993.942469] LustreError: 91233:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2994.088147] Lustre: server umount lustre-MDT0000 complete [ 2996.192144] LustreError: 90875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3004.721438] Lustre: server umount lustre-MDT0001 complete [ 3009.214621] Lustre: server umount MGS complete [ 3023.596756] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:52:53 (1769755973) [ 3045.654858] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 3048.817372] Key type lgssc unregistered [ 3049.263345] LNet: 92976:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3049.270673] LNetError: 92976:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3049.284993] LNet: Removed LNI 192.168.204.123@tcp [ 3050.420949] Key type .llcrypt unregistered [ 3050.431040] Key type ._llcrypt unregistered [ 3069.288084] Key type ._llcrypt registered [ 3069.290762] Key type .llcrypt registered [ 3069.541730] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3071.340631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3071.495169] alg: No test for adler32 (adler32-zlib) [ 3072.943863] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 3073.363820] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 3075.215199] Key type lgssc registered [ 3077.659650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3087.788589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3087.804560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3089.475213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3094.949629] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3099.625052] LustreError: 94182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3099.674048] LustreError: 94182:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3104.750396] LustreError: 94182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3105.035298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3105.724703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3111.071184] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3117.035244] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3121.197914] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3129.855500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3130.349508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3134.477459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3137.030620] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3143.775376] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3151.842681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3151.855251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3151.874315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3154.915997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3154.918518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3154.929160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3160.039357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3160.056783] Lustre: Skipped 1 previous similar message [ 3165.153736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3165.167325] Lustre: Skipped 1 previous similar message [ 3166.175149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3166.239187] LustreError: 96423:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3166.654492] Lustre: server umount lustre-OST0000 complete [ 3172.323635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3172.331228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3172.352360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3172.357289] Lustre: Skipped 1 previous similar message [ 3175.401567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3177.076925] LustreError: 96625:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3177.082706] LustreError: 96625:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3177.227618] Lustre: server umount lustre-MDT0000 complete [ 3180.512317] LustreError: 94178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3180.926022] LustreError: 94163:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756132 with bad export cookie 1869235918678128504 [ 3180.933054] LustreError: 94163:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3180.933333] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3181.090790] LustreError: 96828:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3181.099696] LustreError: 96828:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3181.347652] Lustre: server umount lustre-MDT0001 complete [ 3196.670741] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 3200.068769] Key type lgssc unregistered [ 3200.416567] LNet: 97361:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3200.426194] LNetError: 97361:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3201.447261] LNet: Removed LNI 192.168.204.123@tcp [ 3202.667509] Key type .llcrypt unregistered [ 3202.679337] Key type ._llcrypt unregistered [ 3222.002344] Key type ._llcrypt registered [ 3222.004510] Key type .llcrypt registered [ 3222.086051] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3222.948783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3222.992053] alg: No test for adler32 (adler32-zlib) [ 3224.155101] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 3224.362090] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 3226.015397] Key type lgssc registered [ 3227.024715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3234.828362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3234.833923] LustreError: 98421:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 3234.863446] LustreError: 98425:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 3234.880808] LustreError: 98421:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4962 [ 3234.888476] LustreError: 98421:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3234.903580] LustreError: 98421:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3234.927674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3236.765917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3245.304561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3245.567927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3250.566525] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3256.097701] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing lsmod [ 3259.243558] LustreError: 99593:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 3259.261402] LustreError: 99597:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 3259.266744] LustreError: 99593:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4980 [ 3259.274762] LustreError: 99593:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3259.280545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3259.281933] LustreError: 99593:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3259.701629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3261.756605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3266.018126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3266.024926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.058654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3267.056260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3267.062175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3267.082356] Lustre: Skipped 1 previous similar message [ 3271.205133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3271.221770] Lustre: Skipped 4 previous similar messages [ 3276.258815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3276.270990] Lustre: Skipped 2 previous similar messages [ 3277.279180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3277.370424] LustreError: 99780:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3277.491286] Lustre: server umount lustre-MDT0000 complete [ 3281.379480] LustreError: 98462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3281.397655] LustreError: 98462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3281.404335] LustreError: 98440:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756233 with bad export cookie 4309230330199327640 [ 3281.409187] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3281.437036] LustreError: 98440:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3281.504698] LustreError: 98893:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009b963e3a x1855723852028928/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3281.597675] LustreError: 99989:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3281.604862] LustreError: 99989:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3281.789615] Lustre: server umount lustre-MDT0001 complete [ 3295.772475] LustreError: 100190:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3295.775251] LustreError: 100190:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3296.126135] Lustre: server umount lustre-OST0000 complete [ 3305.472867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3306.144556] LustreError: 100667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3306.250074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3311.588768] LustreError: 100667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3312.098151] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3315.699819] LustreError: 100667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3320.820916] LustreError: 100667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3324.793962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3325.860739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3330.257234] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3338.447609] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3344.712534] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3353.269703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3353.709116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3359.901219] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3366.880804] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3376.353186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3386.857686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3386.859592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3386.866906] Lustre: Skipped 1 previous similar message [ 3391.752866] LustreError: 102933:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3391.761260] LustreError: 102933:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3392.011756] LustreError: 102265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.102940] Lustre: server umount lustre-OST0000 complete [ 3402.210759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3402.236779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3402.239635] Lustre: Skipped 1 previous similar message [ 3404.191909] LustreError: 103135:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3404.206963] LustreError: 103135:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3404.436473] Lustre: server umount lustre-MDT0000 complete [ 3409.657613] LustreError: 103136:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756361 with bad export cookie 4309230330199328571 [ 3409.670111] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3409.683232] LustreError: 103136:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3410.371691] Lustre: server umount lustre-MDT0001 complete [ 3426.785289] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 3431.062559] Key type lgssc unregistered [ 3431.486652] LNet: 103872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3431.492835] LNetError: 103872:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3431.515201] LNet: Removed LNI 192.168.204.123@tcp [ 3432.798085] Key type .llcrypt unregistered [ 3432.807961] Key type ._llcrypt unregistered [ 3463.710170] Key type ._llcrypt registered [ 3463.716594] Key type .llcrypt registered [ 3463.866294] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3465.629556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3465.697441] alg: No test for adler32 (adler32-zlib) [ 3466.816493] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 3467.072937] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 3468.815309] Key type lgssc registered [ 3470.397080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3479.567398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3479.598091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3481.202089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3487.319741] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3490.813539] LustreError: 105270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3490.833450] LustreError: 105270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3495.904520] LustreError: 105270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3497.500099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3497.795131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3502.899269] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3510.607399] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3515.471435] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3523.283188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3523.883491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3525.947427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3530.395424] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3537.973225] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3547.103473] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3552.508416] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3558.883271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3558.886632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3558.901185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3564.005428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3564.021108] Lustre: Skipped 2 previous similar messages [ 3564.304202] LustreError: 107633:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3564.686136] Lustre: server umount lustre-OST0000 complete [ 3569.632683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3569.638280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3569.652948] Lustre: Skipped 1 previous similar message [ 3569.660611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3572.194994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3572.200620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3575.073167] LustreError: 107835:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3575.091101] LustreError: 107835:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3575.466710] Lustre: server umount lustre-MDT0000 complete [ 3577.318463] LustreError: 105269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3577.368783] LustreError: 105269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3580.139528] LustreError: 105249:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756531 with bad export cookie 12115714252171968437 [ 3580.145643] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3580.298659] LustreError: 108036:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3580.318686] LustreError: 108036:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3580.532575] Lustre: server umount lustre-MDT0001 complete [ 3593.743919] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 3597.474444] Key type lgssc unregistered [ 3597.925733] LNet: 108570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3597.948852] LNetError: 108570:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3597.983279] LNet: Removed LNI 192.168.204.123@tcp [ 3598.968105] Key type .llcrypt unregistered [ 3598.970733] Key type ._llcrypt unregistered [ 3622.417237] Key type ._llcrypt registered [ 3622.421159] Key type .llcrypt registered [ 3622.552868] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3623.817719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3623.904147] alg: No test for adler32 (adler32-zlib) [ 3625.290232] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 3625.528489] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 3627.263211] Key type lgssc registered [ 3628.798714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3639.099579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3639.117698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.751143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3645.816878] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3651.048638] LustreError: 109772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3651.072623] LustreError: 109772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3656.162229] LustreError: 109772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3656.352488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3656.948144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3662.395671] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3669.895593] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3675.214909] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3683.319233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3683.961467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3685.935452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3691.186364] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3698.034446] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3711.458066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3711.466269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3711.489537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3712.485743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3712.490389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3712.498060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3713.474638] LustreError: 112038:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3713.773536] Lustre: server umount lustre-OST0000 complete [ 3718.628526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3718.635402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3718.645638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3718.647597] Lustre: Skipped 1 previous similar message [ 3722.723385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3722.724795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3722.748533] Lustre: Skipped 1 previous similar message [ 3724.274513] LustreError: 112240:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3724.284791] LustreError: 112240:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3724.445721] Lustre: server umount lustre-MDT0000 complete [ 3727.857801] LustreError: 109769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3727.900903] LustreError: 109769:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3729.279618] LustreError: 109754:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756680 with bad export cookie 8606352706071889166 [ 3729.286964] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3729.297829] LustreError: 109754:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3729.441982] LustreError: 112441:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3729.447434] LustreError: 112441:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3729.667747] Lustre: server umount lustre-MDT0001 complete [ 3743.734781] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 3746.628462] Key type lgssc unregistered [ 3747.032681] LNet: 112975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3747.054956] LNetError: 112975:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3747.083418] LNet: Removed LNI 192.168.204.123@tcp [ 3748.052373] Key type .llcrypt unregistered [ 3748.054474] Key type ._llcrypt unregistered [ 3775.997800] Key type ._llcrypt registered [ 3776.003922] Key type .llcrypt registered [ 3776.122752] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3777.640804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3777.691959] alg: No test for adler32 (adler32-zlib) [ 3778.705718] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 3778.985092] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 3780.791249] Key type lgssc registered [ 3782.046424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3790.711160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3790.735580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3792.324083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3797.801144] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3801.573667] LustreError: 114374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3801.629591] LustreError: 114374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3806.717285] LustreError: 114374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3810.780052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3811.169903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3817.585287] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3824.988038] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3830.083059] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3838.494664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3839.016237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3846.092446] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3852.141173] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3862.587057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3870.991126] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3875.952473] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3875.957487] Lustre: Skipped 2 previous similar messages [ 3877.048584] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3877.109097] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3881.547383] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3881.557397] Lustre: Skipped 2 previous similar messages [ 3883.812835] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3883.822665] Lustre: Skipped 1 previous similar message [ 3895.177792] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.23@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3895.189467] Lustre: Skipped 2 previous similar messages [ 3902.054569] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.23@tcp 192.168.0.[2,10]@tcp [ 3906.531548] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.23/24@tcp in log lustre-MDT0000 [ 3906.538024] Lustre: Skipped 2 previous similar messages [ 3909.585303] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.23/24@tcp [ 3909.598645] Lustre: Skipped 1 previous similar message [ 3922.400739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3922.406250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3922.416881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3923.936361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3923.946881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.958855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3923.982926] Lustre: Skipped 1 previous similar message [ 3926.352263] LustreError: 117912:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3926.547848] Lustre: server umount lustre-OST0000 complete [ 3934.197784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3934.203906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3934.241898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3934.257106] Lustre: Skipped 1 previous similar message [ 3937.514861] LustreError: 118113:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3937.525141] LustreError: 118113:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3938.029089] Lustre: server umount lustre-MDT0000 complete [ 3939.298784] LustreError: 115267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3939.315440] LustreError: 115267:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3942.023816] LustreError: 117097:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756893 with bad export cookie 143624796973507695 [ 3942.027064] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3942.029362] LustreError: 117097:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3942.187750] LustreError: 118315:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3942.197293] LustreError: 118315:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3942.476931] Lustre: server umount lustre-MDT0001 complete [ 3954.623256] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 3957.567195] Key type lgssc unregistered [ 3957.858628] LNet: 118847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3957.866756] LNetError: 118847:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3957.888393] LNet: Removed LNI 192.168.204.123@tcp [ 3958.830489] Key type .llcrypt unregistered [ 3958.832418] Key type ._llcrypt unregistered [ 3995.720768] Key type ._llcrypt registered [ 3995.722934] Key type .llcrypt registered [ 3995.826746] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3997.167110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3997.327863] alg: No test for adler32 (adler32-zlib) [ 3998.429849] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 3998.652095] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4000.319202] Key type lgssc registered [ 4002.093406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4011.923623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4011.952162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4013.762754] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4013.822262] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.23/24@tcp [ 4013.877814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4020.226504] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4024.291631] LustreError: 120389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4024.325877] LustreError: 120389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4029.411304] LustreError: 120389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4031.427576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4031.810816] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4031.846597] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.23/24@tcp [ 4031.918935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4037.346798] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4043.948753] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4048.849771] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4057.038654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4057.629150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4058.714914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4064.839644] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4071.500321] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4087.268469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4087.269893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4087.302379] Lustre: Skipped 1 previous similar message [ 4087.302845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4087.338035] Lustre: Skipped 1 previous similar message [ 4092.390063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4092.397504] Lustre: Skipped 1 previous similar message [ 4092.615224] LustreError: 122686:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4092.843961] Lustre: server umount lustre-OST0000 complete [ 4097.504141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4097.507191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4097.525900] Lustre: Skipped 1 previous similar message [ 4097.543928] Lustre: Skipped 1 previous similar message [ 4102.626290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4102.635052] Lustre: Skipped 1 previous similar message [ 4103.725129] LustreError: 122889:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4103.740257] LustreError: 122889:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4104.072086] Lustre: server umount lustre-MDT0000 complete [ 4107.748479] LustreError: 120389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4107.777698] LustreError: 120389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4108.764356] LustreError: 122890:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769757060 with bad export cookie 1285442091239721066 [ 4108.773201] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4108.777314] LustreError: 122890:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4108.988325] LustreError: 123091:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4108.999194] LustreError: 123091:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4109.314878] Lustre: server umount lustre-MDT0001 complete [ 4122.169354] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4125.720879] Key type lgssc unregistered [ 4126.042370] LNet: 123624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4126.055589] LNetError: 123624:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4126.079501] LNet: Removed LNI 192.168.204.123@tcp [ 4127.067633] Key type .llcrypt unregistered [ 4127.069884] Key type ._llcrypt unregistered [ 4160.578068] Key type ._llcrypt registered [ 4160.579750] Key type .llcrypt registered [ 4160.659158] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 4178.227254] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4179.568975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4179.614436] alg: No test for adler32 (adler32-zlib) [ 4180.735552] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 4180.960470] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4182.663529] Key type lgssc registered [ 4184.031691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4221.392254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4221.436370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4222.658270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4222.690412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4222.794160] Lustre: lustre-MDT0000: new disk, initializing [ 4222.958440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4222.992810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4228.313684] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4239.184744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4239.275174] Lustre: 126989:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4239.360838] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4239.364799] Lustre: Skipped 1 previous similar message [ 4239.436801] Lustre: lustre-MDT0001: new disk, initializing [ 4239.525369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4239.560415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4239.580320] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4244.203949] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4252.478176] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4257.426822] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4264.430937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4264.796286] Lustre: lustre-OST0000: new disk, initializing [ 4264.804267] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4264.920562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4270.493804] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4271.197515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4271.209575] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4271.333273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4278.243537] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4282.559590] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4282.807586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4288.627730] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4288.998851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4292.065104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4292.074507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4292.090437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4292.100160] Lustre: Skipped 1 previous similar message [ 4297.189585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4297.202776] Lustre: Skipped 1 previous similar message [ 4302.324195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4302.337607] Lustre: Skipped 1 previous similar message [ 4305.375467] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4305.417081] LustreError: 129924:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4305.546703] Lustre: server umount lustre-OST0000 complete [ 4311.523416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4311.538156] LustreError: Skipped 1 previous similar message [ 4311.543054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4311.566113] Lustre: Skipped 1 previous similar message [ 4311.586858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4312.554578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4315.777632] LustreError: 130126:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4315.784055] LustreError: 130126:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4315.947177] Lustre: server umount lustre-MDT0000 complete [ 4317.669326] LustreError: 128045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4317.696675] LustreError: 128045:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4320.063059] LustreError: 126982:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769757271 with bad export cookie 903248063204886112 [ 4320.071199] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4320.085800] LustreError: 126982:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4320.331305] LustreError: 130328:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4320.338168] LustreError: 130328:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4320.518188] Lustre: server umount lustre-MDT0001 complete [ 4329.329209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4329.827619] LustreError: 130805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4329.894093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4334.120856] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4335.073870] LustreError: 130805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4340.207548] LustreError: 130805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4344.260344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4344.507742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4349.353576] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4355.790908] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4360.275346] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4368.166837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4368.535324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4375.489869] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4382.326181] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4387.948447] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4388.219247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4392.733558] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4392.950453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4403.028983] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4403.254326] Lustre: lustre-OST0001: new disk, initializing [ 4403.260714] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4403.372366] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4408.942794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4408.956575] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4409.077316] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4409.293417] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4418.437349] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4418.674830] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4422.649173] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4422.811904] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4427.485436] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff910045e91800.ost_server_uuid 50 [ 4429.748952] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff910045e91800.ost_server_uuid in FULL state after 0 sec [ 4445.159295] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4445.160831] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4445.172176] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4445.172187] Lustre: Skipped 2 previous similar messages [ 4445.180310] Lustre: Skipped 1 previous similar message [ 4445.212105] LustreError: Skipped 1 previous similar message [ 4447.381212] LustreError: 135049:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4447.387306] LustreError: 135049:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4447.435958] Lustre: server umount lustre-OST0001 complete [ 4450.282902] LustreError: 132395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4450.299260] LustreError: 132395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4451.808675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4451.813763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4455.393868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4455.402361] Lustre: Skipped 3 previous similar messages [ 4457.820859] LustreError: 135251:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4457.839609] LustreError: 135251:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4457.911758] Lustre: server umount lustre-OST0000 complete [ 4465.635289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4465.650974] Lustre: Skipped 1 previous similar message [ 4469.074046] LustreError: 135454:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4469.081712] LustreError: 135454:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4469.566205] Lustre: server umount lustre-MDT0000 complete [ 4470.756963] LustreError: 134334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4470.786366] LustreError: 134334:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4475.290422] LustreError: 133805:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769757426 with bad export cookie 903248063204886987 [ 4475.292389] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4475.308731] LustreError: 133805:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4476.281819] Lustre: server umount lustre-MDT0001 complete [ 4500.673614] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4503.916715] Key type lgssc unregistered [ 4504.220464] LNet: 136646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4504.229840] LNetError: 136646:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4504.250370] LNet: Removed LNI 192.168.204.123@tcp [ 4505.148937] Key type .llcrypt unregistered [ 4505.152346] Key type ._llcrypt unregistered [ 4553.654902] Key type ._llcrypt registered [ 4553.657919] Key type .llcrypt registered [ 4553.761917] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 4571.428792] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4573.314594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4573.365933] alg: No test for adler32 (adler32-zlib) [ 4574.423887] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 4574.682055] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4576.415168] Key type lgssc registered [ 4577.958383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4618.281591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4618.390139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4619.809194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4619.861465] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4620.036279] Lustre: lustre-MDT0000: new disk, initializing [ 4620.129931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4620.169901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4624.489965] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4635.773434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4635.867304] Lustre: 140768:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4635.973430] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4635.980642] Lustre: Skipped 1 previous similar message [ 4636.076435] Lustre: lustre-MDT0001: new disk, initializing [ 4636.186642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4636.276437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4636.285145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4640.649460] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4648.149738] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4652.107176] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4658.982593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4659.284509] Lustre: lustre-OST0000: new disk, initializing [ 4659.288279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4659.368528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4664.769578] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4667.526788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4667.533530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4667.737552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4672.233613] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4677.647090] Lustre: Failing over lustre-OST0000 [ 4678.117554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4678.125828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4678.126647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4678.161272] Lustre: Skipped 1 previous similar message [ 4679.743230] LustreError: 143203:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4679.979975] LustreError: 142967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4680.002603] LustreError: 142967:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4680.261642] Lustre: server umount lustre-OST0000 complete [ 4699.709531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4700.274322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4700.301609] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4707.078772] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4710.867382] Lustre: Failing over lustre-MDT0000 [ 4711.060586] LustreError: 144181:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4711.074971] LustreError: 144181:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4711.350406] Lustre: server umount lustre-MDT0000 complete [ 4712.929567] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4712.953710] LustreError: 140774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4712.962729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4712.976237] LustreError: 140774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4712.997225] LustreError: Skipped 1 previous similar message [ 4714.753767] LustreError: 140774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4714.770147] LustreError: 140774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4718.049033] LustreError: 140774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4718.071336] LustreError: 140774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4719.077308] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4724.198600] LustreError: 140780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4724.214318] LustreError: 140780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4729.314402] Lustre: 138991:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769757664/real 1769757664] req@000000005ca6bca2 x1855725267627392/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769757680 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4729.348550] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4731.000724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4734.431863] LustreError: 141820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4734.455638] LustreError: 141820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4739.558235] LustreError: 138990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f0216609 x1855725267632000/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4739.784738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4739.848425] Lustre: lustre-OST0000: Recovery over after 0:20, of 3 clients 3 recovered and 0 were evicted. [ 4739.854809] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4739.873453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4739.959622] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4741.355990] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4743.720602] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4745.201771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4745.207159] Lustre: Skipped 1 previous similar message [ 4745.227927] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4745.298378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4755.432568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4755.435275] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4755.453250] Lustre: Skipped 3 previous similar messages [ 4755.456373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4755.459304] Lustre: Skipped 1 previous similar message [ 4758.337892] LustreError: 145074:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4758.344547] LustreError: 145074:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4758.560974] Lustre: server umount lustre-OST0000 complete [ 4765.672171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4765.672447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4765.682536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4765.682544] Lustre: Skipped 1 previous similar message [ 4765.700097] Lustre: Skipped 1 previous similar message [ 4768.458533] LustreError: 145276:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4768.470023] LustreError: 145276:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4768.669595] Lustre: server umount lustre-MDT0000 complete [ 4770.794393] LustreError: 144934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4770.816572] LustreError: 144934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4772.376546] LustreError: 141547:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769757724 with bad export cookie 14060971752003988381 [ 4772.383338] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4772.389922] LustreError: 141547:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4772.584892] LustreError: 145478:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4772.600648] LustreError: 145478:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4772.816307] Lustre: server umount lustre-MDT0001 complete [ 4784.668081] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4787.178465] Key type lgssc unregistered [ 4787.477665] LNet: 146010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4787.490220] LNetError: 146010:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4787.501220] LNet: Removed LNI 192.168.204.123@tcp [ 4788.213301] Key type .llcrypt unregistered [ 4788.216422] Key type ._llcrypt unregistered [ 4826.167738] Key type ._llcrypt registered [ 4826.169728] Key type .llcrypt registered [ 4826.263449] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4827.946239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4827.991769] alg: No test for adler32 (adler32-zlib) [ 4829.116963] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 4829.367575] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4831.079138] Key type lgssc registered [ 4832.002644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4839.816289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4839.836689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4841.431325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4845.620348] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4851.686887] LustreError: 147512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4851.702925] LustreError: 147512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4854.288365] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4856.813173] LustreError: 147512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4858.457737] LustreError: 147540:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4858.468782] LustreError: 147540:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4858.477878] LustreError: 147540:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4858.626890] LustreError: 148064:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4858.934181] Lustre: server umount lustre-MDT0000 complete [ 4874.387554] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 4882.515733] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4916.752541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4917.063703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4917.100308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4917.181886] Lustre: lustre-MDT0000: new disk, initializing [ 4917.256700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4917.279748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4921.787774] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4932.219765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4932.314954] Lustre: 150694:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4932.362285] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4932.365959] Lustre: Skipped 1 previous similar message [ 4932.428324] Lustre: lustre-MDT0001: new disk, initializing [ 4932.507366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4932.543716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4932.563285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4936.715330] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4944.437721] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4947.983603] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4954.167857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4954.410504] Lustre: lustre-OST0000: new disk, initializing [ 4954.413888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4954.494213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4960.308460] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4961.372899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4961.386101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4961.537780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4967.416482] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4971.229438] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4972.698457] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:25:23 (1769757923) [ 5183.457424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5183.460790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5183.473580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5184.991844] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5184.992138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5185.006801] Lustre: Skipped 1 previous similar message [ 5185.017585] Lustre: Skipped 2 previous similar messages [ 5187.212541] LustreError: 154217:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5187.215612] LustreError: 154217:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5187.295841] Lustre: server umount lustre-MDT0000 complete [ 5189.953288] LustreError: 150688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758141 with bad export cookie 6161010926038004340 [ 5189.957233] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5189.957895] LustreError: 150688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5190.111673] LustreError: 153927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5190.111820] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5190.135993] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5194.143898] LustreError: 153927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5194.146627] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5196.125563] LustreError: 154418:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5196.130820] LustreError: 154418:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5196.269713] Lustre: server umount lustre-MDT0001 complete [ 5205.489671] LustreError: 154621:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5205.499810] LustreError: 154621:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5205.587494] Lustre: server umount lustre-OST0000 complete [ 5213.061598] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 5221.479488] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5247.720414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5247.892476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5247.921948] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5247.981974] Lustre: lustre-MDT0000: new disk, initializing [ 5248.038597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5248.052211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5250.876372] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5258.796997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5258.897749] Lustre: 156949:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5258.940715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5258.943162] Lustre: Skipped 1 previous similar message [ 5258.992131] Lustre: lustre-MDT0001: new disk, initializing [ 5259.069437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5259.096825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5259.108089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5262.603537] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5268.974912] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5271.898415] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5277.250914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5277.453294] Lustre: lustre-OST0000: new disk, initializing [ 5277.459911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5277.549585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5282.897863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5282.909263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5283.069845] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5283.299652] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5289.917499] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5293.262759] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5293.437337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5296.356077] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5296.471358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5298.657334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5298.659571] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5298.672820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5303.673574] LustreError: 159877:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5303.676252] LustreError: 159877:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5303.800703] Lustre: server umount lustre-OST0000 complete [ 5308.898276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5308.900369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5308.906492] Lustre: Skipped 2 previous similar messages [ 5308.913818] Lustre: Skipped 2 previous similar messages [ 5312.799227] LustreError: 160079:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5312.801635] LustreError: 160079:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5312.872183] Lustre: server umount lustre-MDT0000 complete [ 5314.018426] LustreError: 157914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5314.036337] LustreError: 157914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5316.121152] LustreError: 156940:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758267 with bad export cookie 6161010926038163170 [ 5316.130276] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5316.130277] LustreError: 156940:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5316.407511] Lustre: server umount lustre-MDT0001 complete [ 5323.338421] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:31:13 (1769758273) [ 5337.022958] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 5344.826409] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5375.385455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5375.645456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5375.673027] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5375.732752] Lustre: lustre-MDT0000: new disk, initializing [ 5375.796589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5375.819144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5378.976412] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5387.040490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5387.140750] Lustre: 163259:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5387.375649] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5390.738280] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5397.081896] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5400.314942] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5405.540671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5410.366431] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5413.916320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5413.922524] Lustre: Skipped 1 previous similar message [ 5413.931366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5413.990565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5416.690257] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5424.098673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5424.101858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5424.107565] Lustre: Skipped 1 previous similar message [ 5424.119341] Lustre: Skipped 1 previous similar message [ 5427.188937] LustreError: 165798:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5427.195580] LustreError: 165798:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5427.280241] Lustre: server umount lustre-OST0000 complete [ 5433.315547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5435.922308] Lustre: server umount lustre-MDT0000 complete [ 5438.109426] LustreError: 163252:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758389 with bad export cookie 6161010926038164052 [ 5438.111992] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5438.117596] LustreError: 163252:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5438.308994] Lustre: server umount lustre-MDT0001 complete [ 5443.868667] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:33:14 (1769758394) [ 5453.718659] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 5459.951243] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5485.855167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5486.025466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5486.042231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5486.054306] Lustre: Skipped 3 previous similar messages [ 5486.112362] Lustre: lustre-MDT0000: new disk, initializing [ 5486.114638] Lustre: Skipped 2 previous similar messages [ 5486.171818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5486.174737] Lustre: Skipped 2 previous similar messages [ 5486.196200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5488.710250] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5495.793267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5495.855138] Lustre: 169177:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5496.048282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5498.816883] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5504.462424] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5507.157393] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5511.741742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5516.481658] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5519.028395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5522.043213] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5529.569626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5529.571710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5529.576134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5529.582115] Lustre: Skipped 3 previous similar messages [ 5529.586324] Lustre: Skipped 3 previous similar messages [ 5533.044676] LustreError: 171717:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5533.048570] LustreError: 171717:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5533.149583] Lustre: server umount lustre-OST0000 complete [ 5537.248155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5541.539905] Lustre: server umount lustre-MDT0000 complete [ 5544.084846] LustreError: 169169:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758495 with bad export cookie 6161010926038165137 [ 5544.087180] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5544.093778] LustreError: 169169:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5544.308494] Lustre: server umount lustre-MDT0001 complete [ 5551.935382] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5553.824413] Key type lgssc unregistered [ 5554.024672] LNet: 172653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5554.028755] LNetError: 172653:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5554.045207] LNet: Removed LNI 192.168.204.123@tcp [ 5554.554421] Key type .llcrypt unregistered [ 5554.556674] Key type ._llcrypt unregistered [ 5572.196913] Key type ._llcrypt registered [ 5572.198718] Key type .llcrypt registered [ 5572.259253] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5572.824386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5572.842128] alg: No test for adler32 (adler32-zlib) [ 5573.820244] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 5574.017022] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5575.648693] Key type lgssc registered [ 5576.581939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5582.349356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5582.361300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5583.701064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5586.615563] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5593.047400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5593.293974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5596.413865] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5600.812432] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5603.964150] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5609.745400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5610.046564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5614.468688] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5618.582363] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5629.495796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5648.866118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5648.870981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5648.878629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5649.892303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5649.892711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5649.897529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5650.995287] LustreError: 176293:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5651.100309] Lustre: server umount lustre-OST0000 complete [ 5655.011628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5655.018891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5655.021286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5655.021297] Lustre: Skipped 1 previous similar message [ 5659.797274] LustreError: 176495:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5659.803520] LustreError: 176495:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5659.844779] Lustre: server umount lustre-MDT0000 complete [ 5660.128806] LustreError: 174029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5660.145263] LustreError: 174029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5662.575071] LustreError: 174745:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758614 with bad export cookie 979297868387657921 [ 5662.576332] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5662.586672] LustreError: 174745:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5662.785734] LustreError: 176696:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5662.792368] LustreError: 176696:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5662.953773] Lustre: server umount lustre-MDT0001 complete [ 5670.876170] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5672.987279] Key type lgssc unregistered [ 5673.269214] LNet: 177226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5673.276364] LNetError: 177226:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5673.284894] LNet: Removed LNI 192.168.204.123@tcp [ 5673.867321] Key type .llcrypt unregistered [ 5673.872261] Key type ._llcrypt unregistered [ 5693.215853] Key type ._llcrypt registered [ 5693.217157] Key type .llcrypt registered [ 5693.282123] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5694.149711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5694.185238] alg: No test for adler32 (adler32-zlib) [ 5695.136211] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 5695.274738] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5696.895596] Key type lgssc registered [ 5697.614222] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5702.826076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5702.835796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5704.160407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5706.311905] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5711.141492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5711.322988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5713.089186] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5716.072633] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5718.298850] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5722.635463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5722.897463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5726.371346] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5729.072104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5729.980378] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5739.488237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5739.493942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5739.503267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5741.537727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5741.539575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5741.555102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5742.836918] Lustre: lustre-OST0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 5742.844670] Lustre: Skipped 1 previous similar message [ 5745.014096] LustreError: 180853:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5745.096616] Lustre: server umount lustre-OST0000 complete [ 5748.560578] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff91004448f800.ost_server_uuid 50 [ 5749.741595] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5762.016191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5762.023609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5762.027901] Lustre: Skipped 1 previous similar message [ 5762.037583] Lustre: Skipped 1 previous similar message [ 5767.334046] LustreError: 181222:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5767.338220] LustreError: 181222:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5767.394622] Lustre: server umount lustre-MDT0000 complete [ 5768.160868] LustreError: 178615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5768.168739] LustreError: 178615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5769.349511] LustreError: 178598:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758721 with bad export cookie 1330858548437935920 [ 5769.350328] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5769.354917] LustreError: 178598:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5769.465286] LustreError: 181423:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5769.470534] LustreError: 181423:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5769.551117] Lustre: server umount lustre-MDT0001 complete [ 5774.607579] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:38:45 (1769758725) [ 5778.509441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5778.745758] LustreError: 182094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5778.800508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5781.053794] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5784.041288] LustreError: 182094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5786.279516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5786.554815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5789.172171] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5792.309410] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5794.787303] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5799.029410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5799.251541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5802.545912] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5803.377357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5805.805491] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5809.794646] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5809.922929] Lustre: lustre-OST0001: new disk, initializing [ 5809.926405] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5814.015969] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5819.146336] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5819.938460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5819.952912] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5820.022272] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5822.435102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5822.446941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5822.464256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5822.467794] Lustre: Skipped 2 previous similar messages [ 5827.256947] LustreError: 185290:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5827.267936] LustreError: 185290:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5827.341718] Lustre: server umount lustre-OST0000 complete [ 5829.862170] LustreError: 184765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5831.744409] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5831.915541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5834.632567] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5834.766687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5834.978186] LustreError: 183685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5834.998417] LustreError: 183685:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5845.486145] LustreError: 183686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5845.514264] LustreError: 183686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5850.605276] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5850.606709] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5850.621878] LustreError: Skipped 1 previous similar message [ 5850.632989] Lustre: Skipped 1 previous similar message [ 5850.649643] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5850.653901] Lustre: Skipped 5 previous similar messages [ 5853.110159] LustreError: 186015:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5853.115064] LustreError: 186015:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5853.189931] Lustre: server umount lustre-OST0001 complete [ 5858.272130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5858.282173] LustreError: Skipped 1 previous similar message [ 5860.837287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5860.843948] Lustre: Skipped 1 previous similar message [ 5870.049184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5870.135249] LustreError: 186216:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5870.138862] LustreError: 186216:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5870.181231] Lustre: server umount lustre-MDT0000 complete [ 5871.072495] LustreError: 185921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5871.083911] LustreError: 185921:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5872.506216] LustreError: 182074:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758824 with bad export cookie 1330858548437936977 [ 5872.510953] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5872.511170] LustreError: 182074:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5872.699492] Lustre: server umount lustre-MDT0001 complete [ 5888.674932] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:40:39 (1769758839) [ 5893.582326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5893.754873] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5893.783511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5893.932469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5893.935450] Lustre: Skipped 1 previous similar message [ 5896.823429] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5902.227845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5902.296535] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5902.329701] Lustre: 187889:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5905.361806] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5909.526805] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5912.258317] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5917.030466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5917.157166] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5921.872501] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5926.072294] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5926.494275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5931.248940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5931.323614] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5931.536867] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5931.543033] Lustre: Skipped 2 previous similar messages [ 5935.675568] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5939.809321] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5941.731229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5941.739559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5941.749500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5941.753058] Lustre: Skipped 5 previous similar messages [ 5947.771555] LustreError: 191074:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5947.796439] LustreError: 191074:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5947.863767] Lustre: server umount lustre-OST0000 complete [ 5950.697331] LustreError: 191075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5962.217975] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5962.225964] LustreError: Skipped 1 previous similar message [ 5967.589363] Lustre: server umount lustre-OST0001 complete [ 5976.934231] Lustre: server umount lustre-MDT0000 complete [ 5980.232123] LustreError: 187880:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758931 with bad export cookie 1330858548437938356 [ 5980.238048] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5980.242189] LustreError: 187880:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5980.405923] LustreError: 191700:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5980.417730] LustreError: 191700:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5980.552786] Lustre: server umount lustre-MDT0001 complete [ 5997.061366] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:42:27 (1769758947) [ 6008.130216] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 6014.835090] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6037.875846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6038.060882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6038.085471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6038.137302] Lustre: lustre-MDT0000: new disk, initializing [ 6038.266717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6038.294310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6040.600817] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6047.349414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6047.437284] Lustre: 195487:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6047.484633] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6047.487485] Lustre: Skipped 1 previous similar message [ 6047.552216] Lustre: lustre-MDT0001: new disk, initializing [ 6047.667822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6047.681927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6050.233535] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6055.657221] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6057.933330] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6062.417709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6062.570306] Lustre: lustre-OST0000: new disk, initializing [ 6062.575336] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6065.983708] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6071.020900] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6072.343089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6072.359148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6072.452331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6073.780436] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6073.953491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6076.580706] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6076.724083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6082.532346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6082.533972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6082.536265] LustreError: Skipped 1 previous similar message [ 6082.537202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6082.549876] Lustre: Skipped 6 previous similar messages [ 6082.552431] Lustre: Skipped 9 previous similar messages [ 6092.255178] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6092.278125] LustreError: 198416:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6092.282750] LustreError: 198416:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6092.348903] Lustre: server umount lustre-OST0000 complete [ 6100.646888] Lustre: server umount lustre-MDT0000 complete [ 6102.953817] LustreError: 195479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759054 with bad export cookie 1330858548437939924 [ 6102.955713] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6102.960055] LustreError: 195479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6103.155973] Lustre: server umount lustre-MDT0001 complete [ 6108.554125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6108.790504] LustreError: 199294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6108.799713] LustreError: 199294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6111.174036] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6116.572334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6119.444651] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6122.867722] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6124.821950] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6128.780657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6132.563369] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6135.935190] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6138.664854] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6138.774368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6141.168600] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6143.373918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec [ 6147.553084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6147.554884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6147.556978] Lustre: Skipped 6 previous similar messages [ 6147.568390] LustreError: Skipped 1 previous similar message [ 6147.571359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.576601] Lustre: Skipped 3 previous similar messages [ 6150.540070] Lustre: server umount lustre-OST0000 complete [ 6153.961716] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6156.181989] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6182.213609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6182.366667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6182.370330] Lustre: Skipped 5 previous similar messages [ 6185.780736] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6187.495293] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.505887] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6189.100830] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6197.736505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.742220] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6203.000644] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec [ 6211.691146] Lustre: server umount lustre-OST0000 complete [ 6220.156970] Lustre: server umount lustre-MDT0000 complete [ 6222.486294] LustreError: 199273:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759174 with bad export cookie 1330858548437940792 [ 6222.489178] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6222.495803] LustreError: 199273:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6222.594163] LustreError: 204107:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6222.596856] LustreError: 204107:0:(obd_class.h:479:obd_check_dev()) Skipped 31 previous similar messages [ 6222.697097] Lustre: server umount lustre-MDT0001 complete [ 6227.818866] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:46:18 (1769759178) [ 6232.266902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6235.194182] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6237.666577] LustreError: 204777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6237.673626] LustreError: 204777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6240.028522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6242.607315] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6245.957865] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6248.159780] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6252.203356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6255.703126] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6259.029272] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6261.432349] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6261.522358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6263.703535] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6264.868651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 6269.038073] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6269.230414] Lustre: lustre-OST0001: new disk, initializing [ 6269.233552] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6272.772338] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6277.323177] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6278.164780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6278.172985] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6278.204188] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6279.534670] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6279.629637] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6281.676715] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6281.820337] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6283.236233] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6283.239607] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6283.246812] LustreError: Skipped 4 previous similar messages [ 6283.247503] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6283.247509] Lustre: Skipped 6 previous similar messages [ 6283.267018] Lustre: Skipped 5 previous similar messages [ 6285.086304] Lustre: server umount lustre-OST0001 complete [ 6288.542695] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6288.666166] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6291.013898] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6291.118682] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6316.583202] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6320.057946] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6322.156422] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6322.165930] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6322.166832] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6323.273198] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6324.255452] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6331.529666] Lustre: server umount lustre-OST0001 complete [ 6351.300644] Lustre: server umount lustre-OST0000 complete [ 6368.223142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6368.312600] Lustre: server umount lustre-MDT0000 complete [ 6370.417697] LustreError: 205496:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759322 with bad export cookie 1330858548437941744 [ 6370.418541] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6370.427441] LustreError: 205496:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6376.688619] Lustre: server umount lustre-MDT0001 complete [ 6390.417684] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:49:01 (1769759341) [ 6394.325847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6394.469922] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6394.482016] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6397.081131] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6402.249220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6402.314965] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6402.332126] Lustre: 212721:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6404.591253] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6407.385765] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6409.463249] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6413.381231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6413.501953] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6416.653518] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6419.711941] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6430.659879] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6430.704541] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6434.136409] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6436.960127] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6437.798426] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6440.092160] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6440.243156] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6442.221562] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6442.345088] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6444.283708] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff910058e10800.ost_server_uuid 50 [ 6445.069589] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff910058e10800.ost_server_uuid in FULL state after 0 sec [ 6445.588565] Lustre: Permanently deactivating lustre-OST0001 [ 6445.597919] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6447.069555] Lustre: Permanently reactivating lustre-OST0001 [ 6454.807505] Lustre: server umount lustre-OST0001 complete [ 6462.807520] Lustre: server umount lustre-OST0000 complete [ 6479.327206] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6479.375153] LustreError: 217107:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6479.378042] LustreError: 217107:0:(obd_class.h:479:obd_check_dev()) Skipped 45 previous similar messages [ 6479.421960] Lustre: server umount lustre-MDT0000 complete [ 6481.103226] LustreError: 212716:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759432 with bad export cookie 1330858548437943032 [ 6481.109581] LustreError: 212716:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6481.109831] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6481.254232] Lustre: server umount lustre-MDT0001 complete [ 6492.218239] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:50:43 (1769759443) [ 6495.884324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6496.045812] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6496.058714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6496.063204] Lustre: Skipped 5 previous similar messages [ 6496.151785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6496.154051] Lustre: Skipped 9 previous similar messages [ 6498.048864] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6502.204586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6502.260354] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6502.274758] Lustre: 218828:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6504.172179] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6506.687384] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6508.268691] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6511.595360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6511.727372] Lustre: Permanently deactivating lustre-OST0000 [ 6511.737191] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6514.896824] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6514.996655] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6517.545153] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6520.938132] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6522.336709] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6523.964296] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6526.617659] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6527.420633] Lustre: Permanently reactivating lustre-OST0000 [ 6529.229994] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6529.233294] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6529.239217] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6529.243353] Lustre: Skipped 1 previous similar message [ 6529.258310] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6529.270888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6542.276944] Lustre: server umount lustre-OST0001 complete [ 6542.821363] LustreError: 221985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6542.829310] LustreError: 221985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 31 previous similar messages [ 6544.864708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6544.865886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6544.870629] LustreError: Skipped 9 previous similar messages [ 6544.871552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6544.880405] Lustre: Skipped 16 previous similar messages [ 6544.897151] Lustre: Skipped 28 previous similar messages [ 6550.245912] Lustre: server umount lustre-OST0000 complete [ 6566.367105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6566.446044] Lustre: server umount lustre-MDT0000 complete [ 6568.214756] LustreError: 218819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759519 with bad export cookie 1330858548437944684 [ 6568.216183] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6568.226975] LustreError: 218819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6568.392393] Lustre: server umount lustre-MDT0001 complete [ 6572.798636] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 6574.005475] Key type lgssc unregistered [ 6574.170622] LNet: 223212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6574.173906] LNetError: 223212:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6574.183671] LNet: Removed LNI 192.168.204.123@tcp [ 6574.571662] Key type .llcrypt unregistered [ 6574.573253] Key type ._llcrypt unregistered [ 6585.406209] Key type ._llcrypt registered [ 6585.407822] Key type .llcrypt registered [ 6585.455505] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6585.866569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6585.924690] alg: No test for adler32 (adler32-zlib) [ 6586.828989] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 6586.926058] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 6588.519171] Key type lgssc registered [ 6589.051892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6593.601835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6593.610621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.850531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6596.686206] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6601.034393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6601.103708] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6601.121196] Lustre: 224626:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6601.127609] Lustre: Permanently deactivating lustre-MDT0001 [ 6601.141365] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6601.328052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6603.443312] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6603.901974] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6603.908872] LustreError: 224667:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6603.921872] LustreError: 224667:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6603.928567] LustreError: 224667:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 6605.966163] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6607.490433] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6610.620312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6610.774178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6613.172845] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6615.600940] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6618.612333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6618.709161] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6619.756534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6619.799850] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6621.258496] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6623.741854] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6625.578894] Lustre: Permanently reactivating lustre-MDT0001 [ 6625.591168] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6625.595497] Lustre: Skipped 2 previous similar messages [ 6625.896816] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6625.902292] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6633.476758] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6634.304345] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6636.554607] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6636.659902] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6638.519968] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6638.621595] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6639.883805] Lustre: Permanently deactivating lustre-MDT0001 [ 6639.893023] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6639.895542] Lustre: Skipped 2 previous similar messages [ 6647.815689] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6650.856848] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6650.859928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6650.866985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6651.874049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6651.874070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6651.881660] Lustre: Skipped 2 previous similar messages [ 6651.893379] Lustre: Skipped 3 previous similar messages [ 6655.487363] LustreError: 228742:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6655.514973] Lustre: server umount lustre-MDT0000 complete [ 6656.996323] LustreError: 224637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6657.003292] LustreError: 224637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6657.046544] LustreError: 224616:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759608 with bad export cookie 3040609368078819419 [ 6657.050437] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6657.050526] LustreError: 224616:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6657.119739] LustreError: 228943:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6657.122282] LustreError: 228943:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6657.185834] Lustre: server umount lustre-MDT0001 complete [ 6664.876103] LustreError: 229143:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6664.878680] LustreError: 229143:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6664.897769] Lustre: server umount lustre-OST0000 complete [ 6669.162079] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:53:40 (1769759620) [ 6673.561480] LustreError: 230148:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6673.567761] LustreError: 230148:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6673.619397] Lustre: server umount lustre-OST0001 complete [ 6676.172303] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 6680.120388] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6695.094099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6695.229133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6695.235867] Lustre: Skipped 2 previous similar messages [ 6695.252464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6695.297920] Lustre: lustre-MDT0000: new disk, initializing [ 6695.339275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6695.357157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6696.903371] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6701.808847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6701.853360] Lustre: 232316:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6701.877362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6701.879955] Lustre: Skipped 1 previous similar message [ 6701.917909] Lustre: lustre-MDT0001: new disk, initializing [ 6701.965973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6701.970519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6703.698960] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6707.405214] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6708.975347] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6712.218037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6712.343158] Lustre: lustre-OST0000: new disk, initializing [ 6712.348054] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6712.388335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6712.392632] Lustre: Skipped 1 previous similar message [ 6714.869719] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6718.449110] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6720.004415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6720.008688] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6720.059671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6724.493658] LustreError: 232323:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6729.360815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6729.444714] Lustre: lustre-OST0001: new disk, initializing [ 6729.447073] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6729.480502] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6731.837818] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6734.591182] LustreError: 232323:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6734.616680] LustreError: 232323:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6735.437715] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6736.073878] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6736.390598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6736.394108] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6736.422572] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6744.711308] LustreError: 232323:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6746.354214] Lustre: lustre-OST0001: Not available for connect from 192.168.204.23@tcp (stopping) [ 6746.601856] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6746.604045] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6751.459257] Lustre: lustre-OST0001: Not available for connect from 192.168.204.23@tcp (stopping) [ 6751.466588] Lustre: Skipped 2 previous similar messages [ 6751.662843] LustreError: 235802:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6751.665534] LustreError: 235802:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6751.697527] Lustre: server umount lustre-OST0001 complete [ 6751.718704] LustreError: 235715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6751.726542] LustreError: 235715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6756.836330] LustreError: 234041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6756.844595] LustreError: 234041:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6761.957336] LustreError: 234039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6761.964831] LustreError: 234039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6766.048952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6766.049043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6766.051896] LustreError: Skipped 1 previous similar message [ 6766.056687] Lustre: Skipped 1 previous similar message [ 6766.059788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6766.062821] Lustre: Skipped 1 previous similar message [ 6767.074569] LustreError: 235715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6767.081156] LustreError: 235715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6770.354144] LustreError: 236026:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6770.360220] LustreError: 236026:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6770.410889] Lustre: server umount lustre-OST0000 complete [ 6772.193229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6772.202489] Lustre: Skipped 1 previous similar message [ 6777.312329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6777.315218] Lustre: Skipped 5 previous similar messages [ 6778.336204] Lustre: server umount lustre-MDT0000 complete [ 6780.213398] LustreError: 234057:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759731 with bad export cookie 3040609368078822226 [ 6780.213743] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6780.221576] LustreError: 234057:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6786.386902] LustreError: 236429:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6786.390092] LustreError: 236429:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6786.458982] Lustre: server umount lustre-MDT0001 complete [ 6790.989306] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 6792.210298] Key type lgssc unregistered [ 6792.352745] LNet: 236961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6792.357350] LNetError: 236961:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6792.367843] LNet: Removed LNI 192.168.204.123@tcp [ 6792.771249] Key type .llcrypt unregistered [ 6792.772269] Key type ._llcrypt unregistered [ 6809.437013] Key type ._llcrypt registered [ 6809.438470] Key type .llcrypt registered [ 6809.479459] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6809.919270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6809.929552] alg: No test for adler32 (adler32-zlib) [ 6810.825838] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 6810.951137] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 6812.543168] Key type lgssc registered [ 6813.095719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6816.879190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6816.889350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6818.041463] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6818.054228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6818.153860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6820.079981] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6824.398150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6824.448641] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6824.468985] Lustre: 239127:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6824.600329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6826.274107] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6828.730473] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6830.226866] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6833.190318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6833.285474] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6833.397044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6836.034701] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6838.347879] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6841.877261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6887.909115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6887.909548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6887.911185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6887.912955] LustreError: Skipped 1 previous similar message [ 6887.917158] Lustre: Skipped 1 previous similar message [ 6890.351761] LustreError: 241481:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6890.415899] Lustre: server umount lustre-OST0000 complete [ 6892.437553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6897.621453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6897.765860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6900.089945] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6902.281323] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6908.004078] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6908.008452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6908.008460] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6936.546363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6936.551864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6936.554106] Lustre: Skipped 2 previous similar messages [ 6941.665934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6941.668015] Lustre: Skipped 1 previous similar message [ 6942.129245] LustreError: 243143:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6942.131870] LustreError: 243143:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6942.189400] Lustre: server umount lustre-OST0000 complete [ 6946.784963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6946.785904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6946.789743] Lustre: Skipped 2 previous similar messages [ 6951.904336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6951.907344] Lustre: Skipped 2 previous similar messages [ 6958.047130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6958.089907] LustreError: 243345:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6958.092632] LustreError: 243345:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6958.122324] Lustre: server umount lustre-MDT0000 complete [ 6959.725406] LustreError: 243346:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759911 with bad export cookie 381889843289271083 [ 6959.729286] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6959.731279] LustreError: 243346:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6959.862772] Lustre: server umount lustre-MDT0001 complete [ 6964.494811] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 6965.770493] Key type lgssc unregistered [ 6965.942561] LNet: 244079:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6965.945622] LNetError: 244079:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6965.953334] LNet: Removed LNI 192.168.204.123@tcp [ 6966.346056] Key type .llcrypt unregistered [ 6966.347678] Key type ._llcrypt unregistered [ 6977.261959] Key type ._llcrypt registered [ 6977.263506] Key type .llcrypt registered [ 6977.307101] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6977.673495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6977.741185] alg: No test for adler32 (adler32-zlib) [ 6978.631478] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 6978.720955] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 6980.311168] Key type lgssc registered [ 6980.744326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6983.721517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6983.727591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6984.906258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6986.331453] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6990.007054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6990.188368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6991.827918] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6994.390731] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6996.001822] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6998.814402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6998.938657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7001.070875] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7003.287939] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7010.216138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7025.637298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7025.637916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7025.638937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7025.641865] Lustre: Skipped 1 previous similar message [ 7025.645169] LustreError: Skipped 1 previous similar message [ 7028.402106] LustreError: 248488:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7028.459908] Lustre: server umount lustre-OST0000 complete [ 7030.754795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7030.761520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7030.764594] Lustre: Skipped 2 previous similar messages [ 7035.872601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7035.876917] Lustre: Skipped 1 previous similar message [ 7036.289394] LustreError: 248689:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7036.291983] LustreError: 248689:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7036.322228] Lustre: server umount lustre-MDT0000 complete [ 7037.760393] LustreError: 245434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759989 with bad export cookie 10791018774731461450 [ 7037.763403] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7037.764501] LustreError: 245434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7037.836939] LustreError: 248891:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7037.839670] LustreError: 248891:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7037.904352] Lustre: server umount lustre-MDT0001 complete [ 7042.000221] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7043.186997] Key type lgssc unregistered [ 7043.322373] LNet: 249422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7043.325420] LNetError: 249422:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7043.333347] LNet: Removed LNI 192.168.204.123@tcp [ 7043.704126] Key type .llcrypt unregistered [ 7043.705497] Key type ._llcrypt unregistered [ 7052.782650] Key type ._llcrypt registered [ 7052.783834] Key type .llcrypt registered [ 7052.837444] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7053.252756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7053.284100] alg: No test for adler32 (adler32-zlib) [ 7054.186314] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 7054.329508] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7055.935222] Key type lgssc registered [ 7056.402127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7059.656605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7059.666725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7060.863573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7062.383117] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7065.728913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7065.870625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7067.449698] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7069.831381] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7071.385701] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7074.224852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7074.364610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7076.494417] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7078.561260] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7085.671883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7096.290715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7096.291564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7096.295122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7096.299939] Lustre: Skipped 1 previous similar message [ 7099.181229] LustreError: 253010:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7099.218483] Lustre: server umount lustre-OST0000 complete [ 7101.408747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7101.409359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7101.412294] Lustre: Skipped 1 previous similar message [ 7101.416471] Lustre: Skipped 2 previous similar messages [ 7106.528436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7106.530520] Lustre: Skipped 1 previous similar message [ 7106.620832] LustreError: 253213:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7106.622979] LustreError: 253213:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7106.651819] Lustre: server umount lustre-MDT0000 complete [ 7107.966334] LustreError: 250582:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760059 with bad export cookie 3915552197631495104 [ 7107.969504] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7107.970256] LustreError: 250582:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7108.036789] LustreError: 253415:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7108.039145] LustreError: 253415:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7108.088653] Lustre: server umount lustre-MDT0001 complete [ 7112.126521] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7113.207446] Key type lgssc unregistered [ 7113.336436] LNet: 253956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7113.339310] LNetError: 253956:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7113.347767] LNet: Removed LNI 192.168.204.123@tcp [ 7113.653172] Key type .llcrypt unregistered [ 7113.654626] Key type ._llcrypt unregistered [ 7122.806993] Key type ._llcrypt registered [ 7122.808287] Key type .llcrypt registered [ 7122.848301] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7123.257986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7123.264485] alg: No test for adler32 (adler32-zlib) [ 7124.113756] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 7124.194089] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7125.775162] Key type lgssc registered [ 7126.134618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7128.839120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7128.845195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7130.006366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7131.411448] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7134.475484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7134.603121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7136.027281] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7137.896097] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7139.142781] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7141.486935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7141.608537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7143.638508] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7145.715688] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7154.918534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7165.412393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7165.413104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7165.413824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7165.416050] Lustre: Skipped 1 previous similar message [ 7165.418441] LustreError: Skipped 1 previous similar message [ 7170.529290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7170.531840] Lustre: Skipped 1 previous similar message [ 7170.861527] LustreError: 258366:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7170.900919] Lustre: server umount lustre-OST0000 complete [ 7175.649276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7175.649413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7175.649916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7175.659462] Lustre: Skipped 1 previous similar message [ 7178.429763] LustreError: 258569:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7178.431646] LustreError: 258569:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7178.460505] Lustre: server umount lustre-MDT0000 complete [ 7179.732351] LustreError: 258570:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760131 with bad export cookie 5311698526286636898 [ 7179.734860] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7179.736076] LustreError: 258570:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7179.799159] LustreError: 258771:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7179.802030] LustreError: 258771:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7179.848502] Lustre: server umount lustre-MDT0001 complete [ 7183.416402] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7184.505695] Key type lgssc unregistered [ 7184.661349] LNet: 259301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7184.664153] LNetError: 259301:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7184.672330] LNet: Removed LNI 192.168.204.123@tcp [ 7185.002407] Key type .llcrypt unregistered [ 7185.003834] Key type ._llcrypt unregistered [ 7192.420097] Key type ._llcrypt registered [ 7192.421733] Key type .llcrypt registered [ 7192.479110] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7192.888569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7192.938882] alg: No test for adler32 (adler32-zlib) [ 7193.835048] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 7193.937253] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7195.527211] Key type lgssc registered [ 7195.972680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7198.927461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7198.933663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7200.112080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7201.551251] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7204.862133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7204.999959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7206.407179] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7208.567635] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7209.914982] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7212.497244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7212.627184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7214.574451] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7216.361097] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7224.940056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7235.554806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7235.554869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7235.563564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7236.783755] LustreError: 262936:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7236.821328] Lustre: server umount lustre-OST0000 complete [ 7240.672679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7240.678067] Lustre: Skipped 1 previous similar message [ 7240.680630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7240.682994] Lustre: Skipped 2 previous similar messages [ 7244.283133] LustreError: 263138:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7244.285094] LustreError: 263138:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7244.309049] Lustre: server umount lustre-MDT0000 complete [ 7245.623417] LustreError: 260476:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760197 with bad export cookie 9615704383933974891 [ 7245.624069] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7245.630124] LustreError: 260476:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7245.708352] LustreError: 263342:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7245.710747] LustreError: 263342:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7245.782201] Lustre: server umount lustre-MDT0001 complete [ 7249.476367] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7250.566663] Key type lgssc unregistered [ 7250.699402] LNet: 263872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7250.701642] LNetError: 263872:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7250.712352] LNet: Removed LNI 192.168.204.123@tcp [ 7251.031864] Key type .llcrypt unregistered [ 7251.032872] Key type ._llcrypt unregistered [ 7254.643256] Key type ._llcrypt registered [ 7254.644701] Key type .llcrypt registered [ 7254.680240] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7260.171795] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 7266.550283] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7266.931489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7267.021124] alg: No test for adler32 (adler32-zlib) [ 7267.875490] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 7267.969398] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7269.551175] Key type lgssc registered [ 7269.912383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7281.363725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7281.371560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7282.463050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7282.476081] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7282.515511] Lustre: lustre-MDT0000: new disk, initializing [ 7282.538121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7282.546121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7283.888232] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7288.141157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7288.184305] Lustre: 267344:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7288.224296] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7288.226896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7288.230025] Lustre: Skipped 1 previous similar message [ 7288.268560] Lustre: lustre-MDT0001: new disk, initializing [ 7288.301719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7288.314948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7288.318676] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7289.644533] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7292.742841] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7293.996737] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7296.328822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7296.440584] Lustre: lustre-OST0000: new disk, initializing [ 7296.442913] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 7296.443665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7296.484620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7298.329738] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7301.539379] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7301.613515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7301.619916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7301.635063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7302.951474] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7303.027891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7304.431110] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7304.513995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7306.723778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7306.723935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7306.727761] LustreError: Skipped 1 previous similar message [ 7306.732120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7306.735711] Lustre: Skipped 1 previous similar message [ 7307.247776] LustreError: 270274:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7307.291775] Lustre: server umount lustre-OST0000 complete [ 7308.767893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7308.770348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7308.774824] Lustre: Skipped 1 previous similar message [ 7308.776347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7311.840954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7314.687798] LustreError: 270479:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 7314.690480] LustreError: 270479:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7314.720375] Lustre: server umount lustre-MDT0000 complete [ 7315.973395] LustreError: 267336:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760267 with bad export cookie 16477510920731280987 [ 7315.977306] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7315.977803] LustreError: 267336:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7316.045921] LustreError: 270681:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 7316.048062] LustreError: 270681:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7316.108279] Lustre: server umount lustre-MDT0001 complete [ 7318.916556] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:04:30 (1769760270) [ 7321.251766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7321.435874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7322.754992] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7325.663172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7325.783749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7327.128767] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7329.095945] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7330.347344] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7332.764184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7334.782779] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7336.617100] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7356.384438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7356.386469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7356.390964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7356.392846] Lustre: Skipped 2 previous similar messages [ 7361.006075] LustreError: 273614:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7361.008218] LustreError: 273614:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7361.049983] Lustre: server umount lustre-OST0000 complete [ 7362.528412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.533689] Lustre: Skipped 2 previous similar messages [ 7367.647941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7367.650341] Lustre: Skipped 5 previous similar messages [ 7368.574080] LustreError: 273817:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7368.576514] LustreError: 273817:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7368.604829] Lustre: server umount lustre-MDT0000 complete [ 7369.759985] LustreError: 271376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760321 with bad export cookie 16477510920731281869 [ 7369.762881] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7369.763768] LustreError: 271376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7369.862573] Lustre: server umount lustre-MDT0001 complete [ 7373.109790] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7374.076229] Key type lgssc unregistered [ 7374.199313] LNet: 274549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7374.201572] LNetError: 274549:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7374.211280] LNet: Removed LNI 192.168.204.123@tcp [ 7374.496848] Key type .llcrypt unregistered [ 7374.497932] Key type ._llcrypt unregistered [ 7383.095997] Key type ._llcrypt registered [ 7383.097237] Key type .llcrypt registered [ 7383.136793] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 7388.972287] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7389.456657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7389.498975] alg: No test for adler32 (adler32-zlib) [ 7390.363275] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 7390.472499] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7392.055128] Key type lgssc registered [ 7392.485878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7426.863441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7426.887156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7428.234238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7428.278553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7428.404189] Lustre: lustre-MDT0000: new disk, initializing [ 7428.478968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7428.490385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7431.299834] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7436.797841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7436.863820] Lustre: 278313:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7436.895293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7436.897434] Lustre: Skipped 1 previous similar message [ 7436.938220] Lustre: lustre-MDT0001: new disk, initializing [ 7436.977709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7436.995068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7437.003712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7438.727817] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7442.505895] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7443.842787] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7446.453447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7446.571904] Lustre: lustre-OST2710: new disk, initializing [ 7446.574284] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7446.610101] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7449.267643] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7452.928213] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7455.231965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7455.236515] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7455.282881] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7455.680183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7455.774319] Lustre: lustre-OST03e8: new disk, initializing [ 7455.777176] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7455.813900] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7458.135360] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7460.066441] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 7462.479211] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7462.577621] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7463.468440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7463.471793] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7463.508493] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7464.219151] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7464.312454] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7465.884985] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7465.973787] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7467.593894] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7467.693157] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7473.121977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7473.125325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7473.131207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7473.636626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7473.637718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7473.641111] Lustre: Skipped 1 previous similar message [ 7475.090297] LustreError: 282642:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7475.137338] Lustre: server umount lustre-MDT0000 complete [ 7476.679867] LustreError: 278304:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760428 with bad export cookie 3723212196017029146 [ 7476.681176] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7476.684514] LustreError: 278304:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7476.758457] LustreError: 282843:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7476.760951] LustreError: 282843:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7476.829899] Lustre: server umount lustre-MDT0001 complete [ 7488.945955] LustreError: 283044:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7488.948717] LustreError: 283044:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7488.968961] Lustre: server umount lustre-OST2710 complete [ 7500.854290] LustreError: 283246:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7500.856838] LustreError: 283246:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7500.926541] Lustre: server umount lustre-OST03e8 complete [ 7507.772735] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 7510.848655] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7523.407460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7523.518686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7523.532166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7523.577596] Lustre: lustre-MDT0000: new disk, initializing [ 7523.618756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7523.627888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7525.076345] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7530.053628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7530.093373] Lustre: 286027:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7530.269946] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7532.160354] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7536.028776] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7537.912417] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7541.172813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7541.301825] Lustre: lustre-OST0000: new disk, initializing [ 7541.304214] Lustre: Skipped 1 previous similar message [ 7541.307414] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7541.311557] Lustre: Skipped 2 previous similar messages [ 7541.350266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7541.353134] Lustre: Skipped 1 previous similar message [ 7543.766553] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7547.486753] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7548.940184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7548.944230] Lustre: Skipped 1 previous similar message [ 7548.946602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7548.989840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7549.199926] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7549.293389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7550.888467] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7551.001624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7554.018038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7554.019874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7554.021389] LustreError: Skipped 1 previous similar message [ 7554.022317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7554.027149] Lustre: Skipped 2 previous similar messages [ 7554.029236] Lustre: Skipped 2 previous similar messages [ 7557.872088] LustreError: 288955:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7557.874927] LustreError: 288955:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7557.936402] Lustre: server umount lustre-OST0000 complete [ 7561.183945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7561.188365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7561.195632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7561.198440] Lustre: Skipped 1 previous similar message [ 7565.655264] Lustre: server umount lustre-MDT0000 complete [ 7567.035281] LustreError: 286801:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760518 with bad export cookie 3723212196017030819 [ 7567.037592] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7567.041933] LustreError: 286801:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7567.113822] LustreError: 289359:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7567.116022] LustreError: 289359:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 7567.177310] Lustre: server umount lustre-MDT0001 complete [ 7570.265983] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:08:41 (1769760521) [ 7570.850212] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7571.458195] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:08:42 (1769760522) [ 7572.536966] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7582.468224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7582.570985] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7582.581427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7582.648519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7584.033932] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7587.026783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7587.068501] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7587.081777] Lustre: 291451:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7588.496037] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7590.320333] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7591.476817] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7604.817060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7604.889334] Lustre: Denying initial registration attempt from nid 192.168.204.123@tcp, specified as failover [ 7604.892139] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7604.895230] LustreError: 293059:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -99 [ 7604.897274] LustreError: 293059:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 7604.899063] LustreError: 293059:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 7604.918521] Lustre: server umount lustre-OST0000 complete [ 7604.920167] LustreError: 293059:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7607.775870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7607.778784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7607.782643] Lustre: Skipped 1 previous similar message [ 7607.784313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7607.785727] Lustre: Skipped 2 previous similar messages [ 7613.116076] LustreError: 293426:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 7613.117925] LustreError: 293426:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7613.139571] Lustre: server umount lustre-MDT0000 complete [ 7614.296124] LustreError: 291443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760565 with bad export cookie 3723212196017031694 [ 7614.299338] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7614.299908] LustreError: 291443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7614.406054] Lustre: server umount lustre-MDT0001 complete [ 7617.918712] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7618.939453] Key type lgssc unregistered [ 7619.063342] LNet: 294160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7619.065467] LNetError: 294160:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7619.071278] LNet: Removed LNI 192.168.204.123@tcp [ 7619.355246] Key type .llcrypt unregistered [ 7619.356258] Key type ._llcrypt unregistered [ 7622.850040] Key type ._llcrypt registered [ 7622.851225] Key type .llcrypt registered [ 7622.885740] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7623.233751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7623.298113] alg: No test for adler32 (adler32-zlib) [ 7624.164728] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 7624.259867] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7625.847176] Key type lgssc registered [ 7626.218284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7660.384335] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7663.242420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7663.249974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7664.342862] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7664.350489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7664.407085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7665.563529] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7668.279733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7668.313920] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7668.325523] Lustre: 296869:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7668.428900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7669.722773] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7671.524840] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7672.725162] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7685.904944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7685.993520] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7686.081177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7688.077198] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7689.839261] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7690.525109] LustreError: 299135:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7690.560488] Lustre: server umount lustre-OST0000 complete [ 7694.303915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7694.306079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7694.310444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7695.343609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7695.344409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7700.448281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7700.450692] Lustre: Skipped 2 previous similar messages [ 7705.568619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7705.570064] Lustre: Skipped 1 previous similar message [ 7706.079125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7706.112086] LustreError: 299335:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 7706.114082] LustreError: 299335:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7706.137942] Lustre: server umount lustre-MDT0000 complete [ 7707.366979] LustreError: 296859:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760659 with bad export cookie 2402927601172669262 [ 7707.370038] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7707.370376] LustreError: 296859:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7707.433607] LustreError: 299542:0:(obd_class.h:479:obd_check_dev()) Device 7 not setup [ 7707.435269] LustreError: 299542:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7707.481388] Lustre: server umount lustre-MDT0001 complete [ 7710.950506] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7711.958388] Key type lgssc unregistered [ 7712.090376] LNet: 300073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7712.092561] LNetError: 300073:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7712.101496] LNet: Removed LNI 192.168.204.123@tcp [ 7712.400727] Key type .llcrypt unregistered [ 7712.401965] Key type ._llcrypt unregistered [ 7721.090578] Key type ._llcrypt registered [ 7721.091839] Key type .llcrypt registered [ 7721.138737] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7721.485536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7721.494146] alg: No test for adler32 (adler32-zlib) [ 7722.336706] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 7722.406793] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7723.983146] Key type lgssc registered [ 7724.358120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7726.924088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7726.930277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7728.130940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7729.395043] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7732.247192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7732.369579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7733.590899] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7735.289316] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7736.463212] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7738.660854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7738.783257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7740.660380] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7742.460961] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7758.305596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7758.306067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7758.307504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7758.309177] Lustre: Skipped 2 previous similar messages [ 7759.079911] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 7759.082238] Lustre: Skipped 2 previous similar messages [ 7762.041299] LustreError: 303713:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7762.066238] Lustre: server umount lustre-MDT0000 complete [ 7763.320430] LustreError: 301432:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760714 with bad export cookie 16625112232322353029 [ 7763.323551] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7763.324599] LustreError: 301432:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7763.423934] LustreError: 301446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7763.424207] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7763.430299] LustreError: 301446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7763.436378] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7764.192144] LustreError: 301445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7768.543835] LustreError: 301445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7768.544168] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7768.549510] Lustre: Skipped 1 previous similar message [ 7769.470329] LustreError: 303916:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7769.472915] LustreError: 303916:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7769.530319] Lustre: server umount lustre-MDT0001 complete [ 7771.637923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7774.494055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7789.023680] LustreError: 300761:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000eb8ea859 x1855728568523520/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7789.147787] LustreError: 303056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7789.154306] LustreError: 303056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7789.171576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7789.191726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 7790.432377] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7793.189900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7793.316461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7794.340020] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7794.617305] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7796.485546] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7798.436558] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7801.113140] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 7802.251652] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7802.853858] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7804.895947] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7804.899479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7804.905530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7804.906888] Lustre: Skipped 1 previous similar message [ 7808.993893] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7808.994556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7810.160312] LustreError: 306219:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7810.162356] LustreError: 306219:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7810.198672] Lustre: server umount lustre-OST0000 complete [ 7814.112680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7814.113052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7814.113655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7814.113660] Lustre: Skipped 2 previous similar messages [ 7814.121661] Lustre: Skipped 1 previous similar message [ 7817.470091] LustreError: 306421:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7817.471690] LustreError: 306421:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7817.494545] Lustre: server umount lustre-MDT0000 complete [ 7818.706242] LustreError: 301430:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760770 with bad export cookie 16625112232322360183 [ 7818.709387] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7818.710165] LustreError: 301430:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7818.811651] Lustre: server umount lustre-MDT0001 complete [ 7822.041210] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7822.975421] Key type lgssc unregistered [ 7823.099343] LNet: 307152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7823.101326] LNetError: 307152:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7823.107277] LNet: Removed LNI 192.168.204.123@tcp [ 7823.367990] Key type .llcrypt unregistered [ 7823.369224] Key type ._llcrypt unregistered [ 7831.970552] Key type ._llcrypt registered [ 7831.971524] Key type .llcrypt registered [ 7832.003543] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7832.334237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7832.375393] alg: No test for adler32 (adler32-zlib) [ 7833.219190] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 7833.299202] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7834.879182] Key type lgssc registered [ 7835.223487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7837.603845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7837.610392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7838.752100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7839.918374] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7842.540148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7842.652284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7843.855949] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7845.433098] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7846.518324] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7847.427844] LustreError: 308555:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7847.431992] LustreError: 308555:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7847.434918] LustreError: 308555:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 7847.904273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7847.904580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7847.904910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7847.908720] Lustre: Skipped 1 previous similar message [ 7853.024398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7853.026159] Lustre: Skipped 2 previous similar messages [ 7853.562231] LustreError: 309912:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7853.590195] Lustre: server umount lustre-MDT0000 complete [ 7854.717953] LustreError: 308505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760806 with bad export cookie 1417425694234413129 [ 7854.720860] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7854.721164] LustreError: 308505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7854.780416] LustreError: 310116:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7854.782310] LustreError: 310116:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7854.834592] Lustre: server umount lustre-MDT0001 complete [ 7859.257129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7859.345290] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7859.359198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7859.431950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7860.635709] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7863.252314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7863.283544] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7863.293633] Lustre: 310884:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7863.386682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7864.602905] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7866.256537] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7867.416269] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7869.819743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7869.884903] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7869.887763] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7869.890264] LustreError: 312539:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 7869.892268] LustreError: 312539:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 7869.893728] LustreError: 312539:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 7869.914760] Lustre: server umount lustre-OST0000 complete [ 7869.916136] LustreError: 312539:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7872.500255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7872.561280] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7872.615832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7874.366301] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7876.029564] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7877.129540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 7878.420572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7878.449932] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7878.478174] Lustre: lustre-OST0001: new disk, initializing [ 7878.479631] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7880.358350] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7882.197823] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7883.128524] LustreError: 314643:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7883.130016] LustreError: 314643:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7883.145883] Lustre: server umount lustre-OST0001 complete [ 7885.292845] LustreError: 313005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7885.294738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7885.295823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7885.300761] LustreError: 313005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7885.303745] Lustre: Skipped 1 previous similar message [ 7890.347080] LustreError: 314843:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7890.349290] LustreError: 314843:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7890.382039] Lustre: server umount lustre-OST0000 complete [ 7893.983810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7893.985957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7893.989299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7893.991578] Lustre: Skipped 1 previous similar message [ 7897.661742] LustreError: 315053:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup [ 7897.663556] LustreError: 315053:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7897.688518] Lustre: server umount lustre-MDT0000 complete [ 7898.844319] LustreError: 310876:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760850 with bad export cookie 1417425694234413661 [ 7898.846971] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7898.849082] LustreError: 310876:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7898.958674] Lustre: server umount lustre-MDT0001 complete [ 7902.104115] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7903.002233] Key type lgssc unregistered [ 7903.122384] LNet: 315784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7903.125309] LNetError: 315784:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7903.132266] LNet: Removed LNI 192.168.204.123@tcp [ 7903.408780] Key type .llcrypt unregistered [ 7903.410123] Key type ._llcrypt unregistered [ 7932.519225] Key type ._llcrypt registered [ 7932.520468] Key type .llcrypt registered [ 7932.568015] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 7938.438910] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7938.821388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7938.833904] alg: No test for adler32 (adler32-zlib) [ 7939.681176] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 7939.764410] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7941.343122] Key type lgssc registered [ 7941.689631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7952.515951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7952.527933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7953.618196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7953.630552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7953.673420] Lustre: lustre-MDT0000: new disk, initializing [ 7953.698310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7953.705764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7955.038268] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7959.189826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7959.229928] Lustre: 320649:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7959.255920] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7959.257816] Lustre: Skipped 1 previous similar message [ 7959.293123] Lustre: lustre-MDT0001: new disk, initializing [ 7959.326632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7959.339098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7959.342057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7960.572807] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7963.580712] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7964.774630] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7966.968977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7967.070768] Lustre: lustre-OST0000: new disk, initializing [ 7967.073585] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7967.101689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7968.185218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7968.188712] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7968.225085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7968.941495] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7971.878494] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7973.137395] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7973.204347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7974.422523] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7974.496370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7978.464778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7978.465443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7978.467083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7978.467316] LustreError: Skipped 1 previous similar message [ 7978.470577] Lustre: Skipped 1 previous similar message [ 7981.166076] LustreError: 323577:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7981.199276] Lustre: server umount lustre-OST0000 complete [ 7983.584465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7983.585292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7983.588195] Lustre: Skipped 1 previous similar message [ 7983.590491] Lustre: Skipped 1 previous similar message [ 7988.475474] LustreError: 323779:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7988.477378] LustreError: 323779:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7988.506646] Lustre: server umount lustre-MDT0000 complete [ 7989.631329] LustreError: 320640:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760941 with bad export cookie 4416553728589069742 [ 7989.633863] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7989.634687] LustreError: 320640:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7989.703850] LustreError: 323981:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7989.706027] LustreError: 323981:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7989.753791] Lustre: server umount lustre-MDT0001 complete [ 7992.281305] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:15:43 (1769760943) [ 7994.511835] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:15:45 (1769760945) [ 7997.080629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7997.212246] LustreError: 324993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7997.219100] LustreError: 324993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7997.235393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7998.351270] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8000.966399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8002.264172] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8003.916715] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8005.001201] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8007.062319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8007.136565] LustreError: 326586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8007.176637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8007.178824] Lustre: Skipped 1 previous similar message [ 8008.887568] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8010.558566] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8020.177506] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 8020.687534] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8021.189429] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8021.700530] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 8022.295075] Lustre: Failing over lustre-MDT0000 [ 8022.462085] LustreError: 327425:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8022.464365] LustreError: 327425:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8022.489957] Lustre: server umount lustre-MDT0000 complete [ 8026.593654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8026.594078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8026.594431] LustreError: 325702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8026.597111] Lustre: Skipped 1 previous similar message [ 8031.712776] LustreError: 324993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8031.718341] LustreError: 324993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8035.292195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8035.329606] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8035.428731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8035.463966] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8036.776748] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8037.600303] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8040.936019] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8040.948173] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 8040.967459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8042.124521] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8042.663232] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8044.488354] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 8045.169087] Lustre: Failing over lustre-MDT0000 [ 8045.360415] LustreError: 328508:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8045.362205] LustreError: 328508:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8045.387516] Lustre: server umount lustre-MDT0000 complete [ 8046.049497] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8046.051072] LustreError: 324987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8046.055057] Lustre: Skipped 2 previous similar messages [ 8046.060511] LustreError: 324987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8046.066080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8056.289120] LustreError: 324992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8056.296789] LustreError: 324992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 8058.265726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8058.307502] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8058.419160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8058.459768] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8059.768533] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8062.187759] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8063.463959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8063.468014] Lustre: Skipped 2 previous similar messages [ 8063.478174] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 8063.500295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8064.625502] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8065.161164] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8068.576851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8068.578938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8068.580878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8068.583967] Lustre: Skipped 2 previous similar messages [ 8068.586347] Lustre: Skipped 1 previous similar message [ 8073.534884] LustreError: 329531:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8073.537052] LustreError: 329531:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8073.563820] Lustre: server umount lustre-MDT0000 complete [ 8073.696147] LustreError: 325011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8073.701754] LustreError: 325011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8074.786512] LustreError: 324973:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761026 with bad export cookie 4416553728589072395 [ 8074.790762] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8074.906051] Lustre: server umount lustre-MDT0001 complete [ 8085.409152] Lustre: server umount lustre-OST0000 complete [ 8090.532500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8090.727167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8092.046386] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8094.848512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8096.273196] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8098.052306] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8099.201427] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8101.396047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8103.438579] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8105.236702] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8116.903050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8122.337090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8122.338188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8122.339250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8122.339256] Lustre: Skipped 2 previous similar messages [ 8122.346601] Lustre: Skipped 2 previous similar messages [ 8127.458516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8127.460340] Lustre: Skipped 2 previous similar messages [ 8128.300191] LustreError: 333031:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8128.301986] LustreError: 333031:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8128.339397] Lustre: server umount lustre-OST0000 complete [ 8131.040046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8135.765709] Lustre: server umount lustre-MDT0000 complete [ 8137.016043] LustreError: 330748:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761088 with bad export cookie 4416553728589073123 [ 8137.019486] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8137.022607] LustreError: 330748:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8137.138414] Lustre: server umount lustre-MDT0001 complete [ 8140.804580] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8141.900342] Key type lgssc unregistered [ 8142.036336] LNet: 333967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8142.038937] LNetError: 333967:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8142.050331] LNet: Removed LNI 192.168.204.123@tcp [ 8142.377773] Key type .llcrypt unregistered [ 8142.379536] Key type ._llcrypt unregistered [ 8152.203045] Key type ._llcrypt registered [ 8152.204033] Key type .llcrypt registered [ 8152.239818] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8152.626365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8152.669854] alg: No test for adler32 (adler32-zlib) [ 8153.524083] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 8153.607553] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 8155.191097] Key type lgssc registered [ 8155.557258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8158.121300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8158.127085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8159.285870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8160.582944] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8163.312538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8163.434719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8164.728257] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8166.494603] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8167.725860] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8169.894195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8170.015417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8171.894076] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8173.735804] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8184.355167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8194.015818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8194.019130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8194.024411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8194.529500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8194.530726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8194.532982] Lustre: Skipped 1 previous similar message [ 8194.536231] Lustre: Skipped 1 previous similar message [ 8199.037099] LustreError: 337703:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8199.063784] Lustre: server umount lustre-MDT0000 complete [ 8199.648834] LustreError: 335440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8199.655764] LustreError: 335440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8200.255237] LustreError: 337706:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761151 with bad export cookie 8950802630689796030 [ 8200.258568] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8200.259653] LustreError: 337706:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8200.327879] LustreError: 337908:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8200.331582] LustreError: 337908:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8200.384583] Lustre: server umount lustre-MDT0001 complete [ 8211.824191] LustreError: 338110:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8211.826107] LustreError: 338110:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8211.867270] Lustre: server umount lustre-OST0000 complete [ 8216.148319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8216.305102] LustreError: 338839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8216.331076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8217.593103] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8220.428077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8220.547069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8221.839051] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8223.557094] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8224.759628] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8226.880644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8228.812483] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8230.601316] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8241.320486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 8251.359895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8251.362086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8251.366061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8254.331624] LustreError: 341205:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8254.333592] LustreError: 341205:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8254.360538] Lustre: server umount lustre-MDT0000 complete [ 8255.545193] LustreError: 338819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761207 with bad export cookie 8950802630689797129 [ 8255.548610] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8255.549899] LustreError: 338819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8255.653668] Lustre: server umount lustre-MDT0001 complete [ 8267.056530] LustreError: 341607:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8267.058264] LustreError: 341607:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 8267.095479] Lustre: server umount lustre-OST0000 complete [ 8271.954894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8272.105030] LustreError: 342434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8272.129175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8272.131285] Lustre: Skipped 1 previous similar message [ 8273.379391] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8276.124663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8277.550934] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8279.272562] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8280.444890] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8282.619032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8284.540320] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8286.267537] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8297.063381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 8306.655976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8306.659717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8306.663146] Lustre: Skipped 2 previous similar messages [ 8306.664704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8306.666803] Lustre: Skipped 2 previous similar messages [ 8309.101160] LustreError: 344697:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8309.103023] LustreError: 344697:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8309.139493] Lustre: server umount lustre-OST0000 complete [ 8312.287735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8312.290132] LustreError: Skipped 1 previous similar message [ 8312.291915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8312.294811] Lustre: Skipped 1 previous similar message [ 8312.296251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8312.298247] Lustre: Skipped 2 previous similar messages [ 8316.495807] Lustre: server umount lustre-MDT0000 complete [ 8317.678162] LustreError: 342414:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761269 with bad export cookie 8950802630689798137 [ 8317.680934] LustreError: 342414:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8317.681941] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8317.800100] Lustre: server umount lustre-MDT0001 complete [ 8321.497705] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8322.594952] Key type lgssc unregistered [ 8322.732455] LNet: 345632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8322.735072] LNetError: 345632:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8322.745410] LNet: Removed LNI 192.168.204.123@tcp [ 8323.086426] Key type .llcrypt unregistered [ 8323.087466] Key type ._llcrypt unregistered [ 8332.321952] Key type ._llcrypt registered [ 8332.322961] Key type .llcrypt registered [ 8332.355294] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8332.650248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8332.657223] alg: No test for adler32 (adler32-zlib) [ 8333.508567] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 8333.589720] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 8335.167123] Key type lgssc registered [ 8335.508476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8337.934889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8337.940195] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8337.942529] LustreError: 347030:0:(osd_handler.c:8561:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8337.950212] LustreError: 347030:0:(obd_config.c:859:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8337.953553] LustreError: 347030:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8337.956368] LustreError: 347030:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8337.959574] LustreError: 347030:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8340.395160] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8340.397709] LustreError: 347426:0:(osd_handler.c:8561:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8340.406992] LustreError: 347426:0:(obd_config.c:859:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8340.409261] LustreError: 347426:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8340.411408] LustreError: 347426:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8340.413889] LustreError: 347426:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8346.736243] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8347.702804] Key type lgssc unregistered [ 8347.832355] LNet: 348409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8347.834841] LNetError: 348409:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8347.847278] LNet: Removed LNI 192.168.204.123@tcp [ 8348.124640] Key type .llcrypt unregistered [ 8348.125735] Key type ._llcrypt unregistered [ 8353.944968] Key type ._llcrypt registered [ 8353.945938] Key type .llcrypt registered [ 8353.986935] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 8360.307374] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8360.642397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8360.684053] alg: No test for adler32 (adler32-zlib) [ 8361.540806] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 8361.624598] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 8363.199104] Key type lgssc registered [ 8363.570405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8374.878332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8374.884161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8375.974587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8375.984766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8376.021671] Lustre: lustre-MDT0000: new disk, initializing [ 8376.043771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8376.050275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8377.345474] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8381.503846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8381.548528] Lustre: 351500:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8381.577820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8381.579792] Lustre: Skipped 1 previous similar message [ 8381.622524] Lustre: lustre-MDT0001: new disk, initializing [ 8381.661125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8381.673856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8381.678304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8382.940193] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8386.046100] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8387.187952] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8389.334549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8389.441084] Lustre: lustre-OST0000: new disk, initializing [ 8389.444096] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8389.478082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8391.298651] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8394.282487] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8395.591786] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8396.791416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8396.794390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8396.825483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8397.712721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 8399.048923] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8399.128397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8401.891827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8401.894090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8401.894663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8401.894779] LustreError: Skipped 1 previous similar message [ 8401.898634] Lustre: Skipped 1 previous similar message [ 8405.870401] LustreError: 354448:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8405.907364] Lustre: server umount lustre-OST0000 complete [ 8407.519874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8407.523277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8407.529173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8407.531740] Lustre: Skipped 1 previous similar message [ 8412.128416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8412.130042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8413.313401] LustreError: 354649:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8413.315307] LustreError: 354649:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8413.346563] Lustre: server umount lustre-MDT0000 complete [ 8414.644655] LustreError: 351491:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761366 with bad export cookie 5484308254407162282 [ 8414.647371] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8414.649805] LustreError: 351491:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8414.723851] LustreError: 354850:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8414.726028] LustreError: 354850:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8414.784564] Lustre: server umount lustre-MDT0001 complete [ 8417.593134] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8418.152342] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:22:49 (1769761369) [ 8420.242700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8420.407732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8421.632252] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8424.282536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8425.565599] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8427.212572] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8428.332137] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8430.393513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8430.496959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8430.499498] Lustre: Skipped 1 previous similar message [ 8432.196019] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8433.760384] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8435.817388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8435.870437] Lustre: lustre-OST0001: new disk, initializing [ 8435.872436] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8437.546478] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8440.397169] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8443.382441] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8443.384342] Lustre: Skipped 1 previous similar message [ 8448.170366] LustreError: 358745:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 8448.173665] LustreError: 358745:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8448.190625] Lustre: server umount lustre-OST0001 complete [ 8448.483251] LustreError: 357159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8448.487567] LustreError: 357159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8453.604247] LustreError: 357157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8453.609657] LustreError: 357157:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8458.721800] LustreError: 357351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8458.729017] LustreError: 357351:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8460.257481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8460.261833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8460.266426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8460.268276] Lustre: Skipped 2 previous similar messages [ 8462.304117] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8463.843498] LustreError: 358752:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8463.849588] LustreError: 358752:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8466.284218] LustreError: 358980:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8466.286596] LustreError: 358980:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8466.325530] Lustre: server umount lustre-OST0000 complete [ 8468.960686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8468.961611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8468.964127] Lustre: Skipped 2 previous similar messages [ 8468.968319] Lustre: Skipped 4 previous similar messages [ 8473.821100] Lustre: server umount lustre-MDT0000 complete [ 8474.080828] LustreError: 355567:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8474.085892] LustreError: 355567:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8475.085198] LustreError: 355548:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761426 with bad export cookie 5484308254407163150 [ 8475.088405] LustreError: 355548:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8475.088975] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8481.216575] LustreError: 359382:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 8481.219693] LustreError: 359382:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 8481.272589] Lustre: server umount lustre-MDT0001 complete [ 8484.886405] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8485.808237] Key type lgssc unregistered [ 8485.927287] LNet: 359914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8485.929610] LNetError: 359914:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8485.936231] LNet: Removed LNI 192.168.204.123@tcp [ 8486.204599] Key type .llcrypt unregistered [ 8486.205461] Key type ._llcrypt unregistered [ 8502.391058] Key type ._llcrypt registered [ 8502.391985] Key type .llcrypt registered [ 8502.426915] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8502.778756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8502.826556] alg: No test for adler32 (adler32-zlib) [ 8503.684458] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 8503.768937] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 8505.351119] Key type lgssc registered [ 8505.730759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8508.272327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8508.278910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8509.372168] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8509.383730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8509.443480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8510.693531] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8513.388275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8513.420967] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8513.432908] Lustre: 362667:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8513.523459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8514.793283] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8516.466634] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8517.607899] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8518.624029] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8518.626398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8518.630029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8520.672201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8520.672720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8524.419066] LustreError: 364028:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 8524.441965] Lustre: server umount lustre-MDT0000 complete [ 8525.617077] LustreError: 364031:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761477 with bad export cookie 5952401216931724335 [ 8525.620056] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8525.623833] LustreError: 364031:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8525.688763] LustreError: 364232:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 8525.690451] LustreError: 364232:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8525.735102] Lustre: server umount lustre-MDT0001 complete [ 8528.903474] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:24:40 (1769761480) [ 8531.027044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8531.199361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8532.489436] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8535.246387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8535.375748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8536.637736] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8538.382298] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8539.516273] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8541.651077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8541.713956] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8541.777084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8543.568182] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8545.217639] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8554.130933] Lustre: Permanently deactivating lustre-OST0000 [ 8554.138722] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8557.552627] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8558.593627] LustreError: 367548:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8558.596916] LustreError: 367548:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 8558.599039] LustreError: 367548:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8559.163950] LustreError: 367596:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 8559.166069] LustreError: 367596:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8565.805095] LustreError: 367693:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8565.807490] LustreError: 367693:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8565.845743] Lustre: server umount lustre-OST0000 complete [ 8566.713964] LustreError: 367846:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 8566.716515] LustreError: 367846:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8569.312682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8569.313231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8569.316580] Lustre: Skipped 1 previous similar message [ 8569.318392] Lustre: Skipped 1 previous similar message [ 8574.431997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8574.433912] Lustre: Skipped 2 previous similar messages [ 8576.482633] LustreError: 367943:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8576.484890] LustreError: 367943:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8576.509885] Lustre: server umount lustre-MDT0000 complete [ 8577.729173] LustreError: 364933:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761529 with bad export cookie 5952401216931724902 [ 8577.732871] LustreError: 364933:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8577.732958] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8588.259232] LustreError: 368146:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8588.260939] LustreError: 368146:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8588.312942] Lustre: server umount lustre-MDT0001 complete [ 8591.181848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8592.504880] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8593.405089] LustreError: 368986:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 8595.470233] Lustre: 369324:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.123@tcp [ 8596.155096] Lustre: server umount MGS complete [ 8599.807079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8599.957807] LustreError: 370048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8599.962305] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8599.962586] LustreError: 370048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8599.964189] Lustre: Skipped 1 previous similar message [ 8599.983938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8601.219706] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8603.896346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8603.986904] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8605.343907] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8607.332451] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8608.723594] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8609.365156] Lustre: Permanently reactivating lustre-OST0000 [ 8609.370197] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8609.371976] Lustre: Skipped 2 previous similar messages [ 8638.943196] LustreError: 371752:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 8641.537117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8641.658632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8641.660726] Lustre: Skipped 1 previous similar message [ 8643.544144] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8645.346663] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8659.427947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8659.428371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8659.429228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8659.431240] Lustre: Skipped 1 previous similar message [ 8663.532864] LustreError: 373570:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8663.534532] LustreError: 373570:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 8663.572293] Lustre: server umount lustre-OST0000 complete [ 8665.567938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8665.570298] LustreError: Skipped 1 previous similar message [ 8665.571537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8669.665043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8669.666773] Lustre: Skipped 3 previous similar messages [ 8679.391120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8679.471068] Lustre: server umount lustre-MDT0000 complete [ 8679.905307] LustreError: 370048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8679.911252] LustreError: 370048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8680.682247] LustreError: 373778:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761632 with bad export cookie 5952401216931725924 [ 8680.686779] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8680.766557] LustreError: 373980:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8680.769805] LustreError: 373980:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 8680.827090] Lustre: server umount lustre-MDT0001 complete [ 8684.692913] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8685.789884] Key type lgssc unregistered [ 8685.942557] LNet: 374511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8685.946354] LNetError: 374511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8685.957592] LNet: Removed LNI 192.168.204.123@tcp [ 8686.320483] Key type .llcrypt unregistered [ 8686.321641] Key type ._llcrypt unregistered [ 8692.396478] Key type ._llcrypt registered [ 8692.397555] Key type .llcrypt registered [ 8692.437948] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 8698.995203] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8699.339862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8699.382661] alg: No test for adler32 (adler32-zlib) [ 8700.238653] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 8700.320784] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 8701.903244] Key type lgssc registered [ 8702.277795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8714.255734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8714.261190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8715.366568] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8715.379515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8715.422078] Lustre: lustre-MDT0000: new disk, initializing [ 8715.453616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8715.461998] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8716.832537] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8721.065100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8721.110199] Lustre: 377604:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8721.137134] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8721.139851] Lustre: Skipped 1 previous similar message [ 8721.175480] Lustre: lustre-MDT0001: new disk, initializing [ 8721.212743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8721.227850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8721.231279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8722.581435] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8725.673710] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8726.931760] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8729.239470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8729.339574] Lustre: lustre-OST0000: new disk, initializing [ 8729.341398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8729.369661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8729.884169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8729.887329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8729.946044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8731.325888] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8734.474386] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8735.917534] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8736.005346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8737.395620] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8737.485592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8740.325742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8740.326926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8740.328078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8740.329875] Lustre: Skipped 1 previous similar message [ 8744.305145] LustreError: 380531:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 8744.339587] Lustre: server umount lustre-OST0000 complete [ 8746.976178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8746.980874] LustreError: Skipped 1 previous similar message [ 8746.983543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8746.990942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8746.994071] Lustre: Skipped 1 previous similar message [ 8750.560475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8750.561053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8751.806575] LustreError: 380732:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8751.808759] LustreError: 380732:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8751.838313] Lustre: server umount lustre-MDT0000 complete [ 8753.078710] LustreError: 377596:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761704 with bad export cookie 904618285681793353 [ 8753.082702] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8753.085071] LustreError: 377596:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8753.156560] LustreError: 380934:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 8753.158721] LustreError: 380934:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8753.226580] Lustre: server umount lustre-MDT0001 complete [ 8756.146347] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8756.764190] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:28:28 (1769761708) [ 8759.074730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8759.258927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8760.617203] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8763.501158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8763.622289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8764.951088] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8766.693722] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8767.927220] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8770.165775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8772.236976] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8774.095871] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8774.747237] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8785.705292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8788.962575] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8788.962705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8788.964576] LustreError: Skipped 1 previous similar message [ 8788.965105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8788.968371] Lustre: Skipped 1 previous similar message [ 8788.972209] Lustre: Skipped 2 previous similar messages [ 8793.263075] LustreError: 384106:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8793.266620] LustreError: 384106:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8793.303304] Lustre: server umount lustre-OST0000 complete [ 8794.591922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8794.594512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8794.598539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8800.700503] LustreError: 384308:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8800.702430] LustreError: 384308:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8800.723143] Lustre: server umount lustre-MDT0000 complete [ 8801.938181] LustreError: 383264:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761753 with bad export cookie 904618285681794172 [ 8801.941302] LustreError: 383264:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8801.941699] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8802.059687] Lustre: server umount lustre-MDT0001 complete [ 8805.955709] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8807.077985] Key type lgssc unregistered [ 8807.209453] LNet: 385040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8807.212650] LNetError: 385040:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8807.221319] LNet: Removed LNI 192.168.204.123@tcp [ 8807.552179] Key type .llcrypt unregistered [ 8807.553413] Key type ._llcrypt unregistered [ 8815.425833] Key type ._llcrypt registered [ 8815.426996] Key type .llcrypt registered [ 8815.468124] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8815.573389] Key type .llcrypt unregistered [ 8815.574344] Key type ._llcrypt unregistered [ 8822.835960] Key type ._llcrypt registered [ 8822.837165] Key type .llcrypt registered [ 8822.877076] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8823.325497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8823.334057] alg: No test for adler32 (adler32-zlib) [ 8824.187608] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 8824.266103] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 8825.839124] Key type lgssc registered [ 8826.199928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8828.669322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8828.674515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8829.827081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8831.027101] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8833.751251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8833.862625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8835.667516] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8837.404359] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8839.136992] LustreError: 387914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8839.140938] LustreError: 387914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8839.583763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8839.713170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8840.744114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8840.952290] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8844.768159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8844.771186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8844.776334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8845.793134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8845.793234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8845.795118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8845.800890] Lustre: Skipped 1 previous similar message [ 8849.838082] LustreError: 389238:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 8849.873131] Lustre: server umount lustre-OST0000 complete [ 8855.008034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8855.011188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8855.015056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8857.211273] LustreError: 389439:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8857.212869] LustreError: 389439:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8857.232845] Lustre: server umount lustre-MDT0000 complete [ 8858.432449] LustreError: 387168:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761810 with bad export cookie 17301103142060413321 [ 8858.435738] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8858.437023] LustreError: 387168:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8858.506623] LustreError: 389640:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8858.510742] LustreError: 389640:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8858.559527] Lustre: server umount lustre-MDT0001 complete [ 8862.105683] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8863.100777] Key type lgssc unregistered [ 8863.220611] LNet: 390171:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8863.223255] LNetError: 390171:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8863.233281] LNet: Removed LNI 192.168.204.123@tcp [ 8863.538633] Key type .llcrypt unregistered [ 8863.539649] Key type ._llcrypt unregistered [ 8872.212084] Key type ._llcrypt registered [ 8872.212988] Key type .llcrypt registered [ 8872.251463] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8872.665686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8872.673285] alg: No test for adler32 (adler32-zlib) [ 8873.516699] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 8873.597957] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 8875.175148] Key type lgssc registered [ 8875.527693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8877.971787] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8877.978927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8893.407278] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8898.527146] LustreError: 391517:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.123@tcp: failed processing log, type 4: rc = -110 [ 8925.151178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8927.033727] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8928.855063] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8931.089544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8931.261338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8931.280091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8932.612767] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8935.395457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8935.535391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8936.890088] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8938.649459] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8939.781373] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8945.122929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8945.127091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8945.129618] Lustre: Skipped 1 previous similar message [ 8948.014734] LustreError: 393795:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 8948.055787] Lustre: server umount lustre-OST0000 complete [ 8950.240697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8950.241454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8950.244594] Lustre: Skipped 2 previous similar messages [ 8950.247485] Lustre: Skipped 1 previous similar message [ 8955.360077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8955.363093] Lustre: Skipped 1 previous similar message [ 8955.452387] LustreError: 393997:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 8955.454169] LustreError: 393997:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8955.478638] Lustre: server umount lustre-MDT0000 complete [ 8956.724232] LustreError: 391525:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761908 with bad export cookie 13357677070243873993 [ 8956.727961] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8956.797374] LustreError: 394199:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8956.800334] LustreError: 394199:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8956.855603] Lustre: server umount lustre-MDT0001 complete [ 8960.693942] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8961.753036] Key type lgssc unregistered [ 8961.882411] LNet: 394730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8961.885288] LNetError: 394730:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8961.896323] LNet: Removed LNI 192.168.204.123@tcp [ 8962.202428] Key type .llcrypt unregistered [ 8962.203503] Key type ._llcrypt unregistered [ 8971.147356] Key type ._llcrypt registered [ 8971.148741] Key type .llcrypt registered [ 8971.183818] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8971.500635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8971.552061] alg: No test for adler32 (adler32-zlib) [ 8972.405053] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 8972.493312] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 8974.071152] Key type lgssc registered [ 8974.431425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8976.999667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8977.008067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8978.162652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8979.393306] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8982.135819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8982.257322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8983.535215] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8985.298697] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8986.586159] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8988.802213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8988.910669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8990.764168] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8992.527769] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9003.239138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9013.215907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9013.218739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9013.222574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9013.728902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9013.730228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9013.733523] Lustre: Skipped 1 previous similar message [ 9013.735230] Lustre: Skipped 1 previous similar message [ 9015.296209] LustreError: 398368:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9015.325107] Lustre: server umount lustre-MDT0000 complete [ 9016.040827] LustreError: 397721:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.23@tcp arrived at 1769761967 with bad export cookie 4867078898528943969 [ 9016.046600] LustreError: 397721:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9016.647254] LustreError: 397721:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761968 with bad export cookie 4867078898528943346 [ 9016.651187] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9017.824237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9017.827683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9017.833757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9018.847808] LustreError: 396825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9018.851864] LustreError: 396825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9021.855808] LustreError: 396825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9021.856144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9021.859685] LustreError: 396825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9021.861230] Lustre: Skipped 1 previous similar message [ 9022.701512] LustreError: 398570:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9022.703591] LustreError: 398570:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9022.737314] Lustre: server umount lustre-OST0000 complete [ 9024.905739] LustreError: 398923:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9024.907994] LustreError: 398923:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9024.966879] Lustre: server umount lustre-MDT0001 complete [ 9028.609214] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 9029.542273] Key type lgssc unregistered [ 9029.660357] LNet: 399454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9029.663903] LNetError: 399454:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9029.677310] LNet: Removed LNI 192.168.204.123@tcp [ 9029.991664] Key type .llcrypt unregistered [ 9029.992712] Key type ._llcrypt unregistered [ 9039.119463] Key type ._llcrypt registered [ 9039.120467] Key type .llcrypt registered [ 9039.157879] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 9039.472471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9039.499065] alg: No test for adler32 (adler32-zlib) [ 9040.340655] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 9040.421328] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 9041.999113] Key type lgssc registered [ 9042.322713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9044.744277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9044.750152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9045.900862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9047.123200] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9049.872707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9049.986181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9051.241285] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9053.002029] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9054.210262] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9056.489823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9056.609368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9058.452481] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9060.281408] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9070.947168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9080.799858] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9080.802183] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9080.806872] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9081.314467] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9081.314955] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9081.316306] Lustre: Skipped 1 previous similar message [ 9082.939386] LustreError: 403089:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9082.962239] Lustre: server umount lustre-MDT0001 complete [ 9086.432329] LustreError: 402198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9086.432482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9086.433552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9086.434431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9086.437501] LustreError: 402198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9090.671659] LustreError: 403295:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9090.675904] LustreError: 403295:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9090.715361] Lustre: server umount lustre-OST0000 complete [ 9091.552791] LustreError: 402198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9091.946418] LustreError: 403496:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9091.949102] LustreError: 403496:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9092.001572] Lustre: server umount lustre-MDT0000 complete [ 9096.390210] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 9097.287896] Key type lgssc unregistered [ 9097.404375] LNet: 404179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9097.407809] LNetError: 404179:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9097.416300] LNet: Removed LNI 192.168.204.123@tcp [ 9097.705485] Key type .llcrypt unregistered [ 9097.706475] Key type ._llcrypt unregistered [ 9105.728278] Key type ._llcrypt registered [ 9105.729942] Key type .llcrypt registered [ 9105.767277] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 9111.815213] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 9112.104750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9112.111229] alg: No test for adler32 (adler32-zlib) [ 9112.957583] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 9113.036788] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 9114.615102] Key type lgssc registered [ 9114.955072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9126.384957] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 9131.435820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9131.440886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9132.530352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9132.542034] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9132.578445] Lustre: lustre-MDT0000: new disk, initializing [ 9132.601447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9132.607884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9133.847869] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9139.557050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9139.679482] Lustre: lustre-OST0000: new disk, initializing [ 9139.681333] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9139.682867] Lustre: Skipped 1 previous similar message [ 9139.710365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9141.581716] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9144.625908] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9148.411800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9148.415135] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9148.422387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9155.317949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9155.357288] Lustre: 408111:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9155.386115] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9155.423541] Lustre: lustre-MDT0001: new disk, initializing [ 9155.479220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9155.491979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 9155.495022] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 9156.690246] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9160.108733] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9160.186425] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9161.459464] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9161.527181] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9167.591448] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.23@tcp (stopping) [ 9167.841389] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9167.842519] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9169.148656] LustreError: 411521:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9169.173378] Lustre: server umount lustre-MDT0001 complete [ 9178.082153] LustreError: 408118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9178.090121] LustreError: 408118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9183.200754] LustreError: 408121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9183.200824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9183.207702] Lustre: Skipped 1 previous similar message [ 9183.209214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9183.210984] Lustre: Skipped 2 previous similar messages [ 9187.501118] LustreError: 411794:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 9187.503192] LustreError: 411794:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9187.540408] Lustre: server umount lustre-OST0000 complete [ 9188.320807] LustreError: 408121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9203.167094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9203.205107] LustreError: 411995:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 9203.206770] LustreError: 411995:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9203.260795] Lustre: server umount lustre-MDT0000 complete [ 9207.652641] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 9208.605307] Key type lgssc unregistered [ 9208.718337] LNet: 412680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9208.720500] LNetError: 412680:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9208.730261] LNet: Removed LNI 192.168.204.123@tcp [ 9209.030514] Key type .llcrypt unregistered [ 9209.031470] Key type ._llcrypt unregistered [ 9221.224074] Key type ._llcrypt registered [ 9221.225077] Key type .llcrypt registered [ 9221.258398] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 9221.358233] Key type .llcrypt unregistered [ 9221.359278] Key type ._llcrypt unregistered [ 9228.018226] Key type ._llcrypt registered [ 9228.019252] Key type .llcrypt registered [ 9228.060095] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 9228.413412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9228.425881] alg: No test for adler32 (adler32-zlib) [ 9229.307050] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 9229.415114] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 9231.007218] Key type lgssc registered [ 9231.427142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9240.025658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9240.030549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9241.117090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9241.127180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9241.164884] Lustre: lustre-MDT0000: new disk, initializing [ 9241.187540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9241.194094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9242.408978] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9246.523505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9246.568220] Lustre: 415988:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9246.592572] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9246.595504] Lustre: Skipped 1 previous similar message [ 9246.637616] Lustre: lustre-MDT0001: new disk, initializing [ 9246.669696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9246.681098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9246.683925] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9247.929684] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9250.999141] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9252.202478] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9254.493181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9254.591129] Lustre: lustre-OST0000: new disk, initializing [ 9254.592808] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9254.621232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9256.428461] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9257.717713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9257.720412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9257.749053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9259.457472] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9262.047748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9262.049816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9262.053416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9263.072335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9263.074349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9267.837240] LustreError: 418419:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9267.869575] Lustre: server umount lustre-MDT0000 complete [ 9269.064091] LustreError: 415978:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762220 with bad export cookie 14034781177649054200 [ 9269.067122] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9269.069898] LustreError: 415978:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9269.131910] LustreError: 418622:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9269.133668] LustreError: 418622:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9269.180600] Lustre: server umount lustre-MDT0001 complete [ 9280.565347] LustreError: 418822:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9280.567436] LustreError: 418822:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9280.609874] Lustre: server umount lustre-OST0000 complete [ 9284.052195] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 03:37:15 (1769762235) [ 9286.545019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9286.701974] LustreError: 419640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9286.707722] LustreError: 419640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9286.726606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9287.917637] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9290.611749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9292.015221] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9293.760270] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9294.936780] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9308.157923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9308.248915] Lustre: Found index 0 for lustre-OST0000, updating log [ 9308.330377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9308.332460] Lustre: Skipped 1 previous similar message [ 9310.204526] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9311.397810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 9311.933069] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9316.129135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9316.130835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9316.133591] Lustre: Skipped 1 previous similar message [ 9316.135494] Lustre: Skipped 2 previous similar messages [ 9319.148329] LustreError: 421884:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9319.150083] LustreError: 421884:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9319.185841] Lustre: server umount lustre-OST0000 complete [ 9333.728785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9333.729311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9333.731734] Lustre: Skipped 1 previous similar message [ 9333.736203] Lustre: Skipped 3 previous similar messages [ 9338.848130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9338.850058] Lustre: Skipped 1 previous similar message [ 9346.015079] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9346.051197] LustreError: 422147:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9346.052762] LustreError: 422147:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9346.080441] Lustre: server umount lustre-MDT0000 complete [ 9347.278578] LustreError: 420357:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762298 with bad export cookie 14034781177649055579 [ 9347.283115] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9347.414126] Lustre: server umount lustre-MDT0001 complete [ 9350.193177] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 03:38:21 (1769762301) [ 9365.599257] LNet: 1 local NIs in recovery (showing 1): 192.168.204.123@tcp [ 9929.525687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9929.673973] LustreError: 425707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9929.695169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9930.848262] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9933.475674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9933.587812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9934.752099] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9936.409558] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9937.534683] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9950.675809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9950.761779] Lustre: Found index 0 for lustre-OST0000, updating log [ 9950.854202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9952.657562] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9954.414542] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9955.079128] LustreError: 428008:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9955.081091] LustreError: 428008:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9955.116544] Lustre: server umount lustre-OST0000 complete [ 9959.391747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9959.394026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9959.397750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9959.399234] Lustre: Skipped 4 previous similar messages [ 9960.416675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9960.417340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9962.386651] Lustre: server umount lustre-MDT0000 complete [ 9963.556312] LustreError: 425687:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762915 with bad export cookie 14034781177649056636 [ 9963.560055] LustreError: 425687:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9963.560443] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9963.671680] Lustre: server umount lustre-MDT0001 complete [ 9980.927992] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 9985.094030] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 9992.287199] LNet: 1 local NIs in recovery (showing 1): 192.168.204.123@tcp [ 9996.192077] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 03:49:07 (1769762947) [10001.778726] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [10002.818703] Key type lgssc unregistered [10013.215801] LNet: 431959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10013.218415] LNetError: 431959:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10081.829266] LNet: Removed LNI 192.168.204.123@tcp [10082.106140] Key type .llcrypt unregistered [10082.107728] Key type ._llcrypt unregistered [10088.527021] Key type ._llcrypt registered [10088.528624] Key type .llcrypt registered [10088.568717] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [10088.971691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10089.019192] alg: No test for adler32 (adler32-zlib) [10089.881254] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [10089.975849] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [10091.559116] Key type lgssc registered [10091.956699] Lustre: Echo OBD driver; http://www.lustre.org/ [10093.449169] LNet: Added LNI 192.168.204.123@tcp100 [8/256/0/180] [10093.773823] LNet: Added LNI 192.168.204.123@tcp101 [8/256/0/180] [10094.066792] LNet: Added LNI 192.168.204.123@tcp102 [8/256/0/180] [10094.360380] LNet: Added LNI 192.168.204.123@tcp103 [8/256/0/180] [10094.702945] LNet: Added LNI 192.168.204.123@tcp104 [8/256/0/180] [10094.990070] LNet: Added LNI 192.168.204.123@tcp105 [8/256/0/180] [10095.268530] LNet: Added LNI 192.168.204.123@tcp106 [8/256/0/180] [10095.616376] LNet: Added LNI 192.168.204.123@tcp107 [8/256/0/180] [10095.969709] LNet: Added LNI 192.168.204.123@tcp108 [8/256/0/180] [10096.246503] LNet: Added LNI 192.168.204.123@tcp109 [8/256/0/180] [10096.568926] LNet: Added LNI 192.168.204.123@tcp110 [8/256/0/180] [10096.874215] LNet: Added LNI 192.168.204.123@tcp111 [8/256/0/180] [10097.145075] LNet: Added LNI 192.168.204.123@tcp112 [8/256/0/180] [10097.417668] LNet: Added LNI 192.168.204.123@tcp113 [8/256/0/180] [10097.684897] LNet: Added LNI 192.168.204.123@tcp114 [8/256/0/180] [10098.065397] LNet: Added LNI 192.168.204.123@tcp115 [8/256/0/180] [10098.399661] LNet: Added LNI 192.168.204.123@tcp116 [8/256/0/180] [10098.674251] LNet: Added LNI 192.168.204.123@tcp117 [8/256/0/180] [10098.986648] LNet: Added LNI 192.168.204.123@tcp118 [8/256/0/180] [10099.336465] LNet: Added LNI 192.168.204.123@tcp119 [8/256/0/180] [10099.633865] LNet: Added LNI 192.168.204.123@tcp120 [8/256/0/180] [10099.940512] LNet: Added LNI 192.168.204.123@tcp121 [8/256/0/180] [10100.257855] LNet: Added LNI 192.168.204.123@tcp122 [8/256/0/180] [10100.585157] LNet: Added LNI 192.168.204.123@tcp123 [8/256/0/180] [10100.880804] LNet: Added LNI 192.168.204.123@tcp124 [8/256/0/180] [10101.192728] LNet: Added LNI 192.168.204.123@tcp125 [8/256/0/180] [10101.571763] LNet: Added LNI 192.168.204.123@tcp126 [8/256/0/180] [10101.863284] LNet: Added LNI 192.168.204.123@tcp127 [8/256/0/180] [10102.158041] LNet: Added LNI 192.168.204.123@tcp128 [8/256/0/180] [10102.478359] LNet: Added LNI 192.168.204.123@tcp129 [8/256/0/180] [10102.782033] LNet: Added LNI 192.168.204.123@tcp130 [8/256/0/180] [10103.064898] LNet: Added LNI 192.168.204.123@tcp131 [8/256/0/180] [10103.348533] LNet: Added LNI 192.168.204.123@tcp132 [8/256/0/180] [10103.641385] LNet: Added LNI 192.168.204.123@tcp133 [8/256/0/180] [10103.971065] LNet: Added LNI 192.168.204.123@tcp134 [8/256/0/180] [10104.331986] LNet: Added LNI 192.168.204.123@tcp135 [8/256/0/180] [10106.728299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10106.734213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10107.792691] Lustre: 434939:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.123@tcp: can't add NID 192.168.204.123@tcp130: rc = -75 [10107.886091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10107.900654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10107.943903] Lustre: lustre-MDT0000: new disk, initializing [10107.986927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10107.997489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10109.364136] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10113.503087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10113.563938] Lustre: 434957:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10113.589826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10113.591329] Lustre: Skipped 1 previous similar message [10113.626869] Lustre: lustre-MDT0001: new disk, initializing [10113.638763] Lustre: 435740:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.123@tcp: can't add NID 192.168.204.123@tcp130: rc = -75 [10113.641139] Lustre: 435740:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [10113.689667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10113.701250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10113.706144] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10114.905361] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10115.888702] Lustre: 436088:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.123@tcp: can't add NID 192.168.204.123@tcp130: rc = -75 [10115.891079] Lustre: 436088:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [10118.367867] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10119.712201] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10122.089876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10122.233024] Lustre: lustre-OST0000: new disk, initializing [10122.234685] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10122.273099] Lustre: 436708:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.123@tcp: can't add NID 192.168.204.123@tcp130: rc = -75 [10122.275351] Lustre: 436708:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [10122.297201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10124.110927] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10124.927933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10124.932403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10124.971182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10127.126633] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10130.404204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10130.404942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10130.406384] LustreError: Skipped 1 previous similar message [10130.407169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10130.409487] Lustre: Skipped 1 previous similar message [10135.474137] LustreError: 437540:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10135.542546] Lustre: server umount lustre-OST0000 complete [10139.615855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10139.617870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10139.621427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10139.622959] Lustre: Skipped 1 previous similar message [10140.640662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10140.640672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10145.760866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10145.765243] Lustre: Skipped 2 previous similar messages [10150.881322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10150.883613] Lustre: Skipped 1 previous similar message [10151.391091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10151.428555] LustreError: 437741:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10151.430427] LustreError: 437741:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10151.459523] Lustre: server umount lustre-MDT0000 complete [10152.727130] LustreError: 436702:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769763104 with bad export cookie 5129751831236416397 [10152.733054] LustreError: 436702:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10152.735369] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10152.832665] LustreError: 437945:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10152.835962] LustreError: 437945:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10152.912173] Lustre: server umount lustre-MDT0001 complete [10156.398711] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [10157.486488] Key type lgssc unregistered [10157.640779] LNet: 438484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10157.644649] LNetError: 438484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10157.647852] LNet: Removed LNI 192.168.204.123@tcp [10158.687257] LNet: Removed LNI 192.168.204.123@tcp100 [10158.689852] LNetError: 438484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10158.694240] LNetError: 438484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10158.698992] LNet: Removed LNI 192.168.204.123@tcp101 [10159.711177] LNet: Removed LNI 192.168.204.123@tcp102 [10159.712274] LNetError: 438484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10159.713944] LNetError: 438484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10159.715798] LNet: Removed LNI 192.168.204.123@tcp103 [10160.735095] LNet: Removed LNI 192.168.204.123@tcp104 [10161.759098] LNet: Removed LNI 192.168.204.123@tcp105 [10161.760397] LNetError: 438484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10161.762276] LNetError: 438484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [10161.764628] LNet: Removed LNI 192.168.204.123@tcp106 [10162.783134] LNet: Removed LNI 192.168.204.123@tcp107 [10163.807091] LNet: Removed LNI 192.168.204.123@tcp108 [10164.831103] LNet: Removed LNI 192.168.204.123@tcp109 [10165.855162] LNet: Removed LNI 192.168.204.123@tcp110 [10165.856302] LNetError: 438484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10165.858013] LNetError: 438484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [10165.859809] LNet: Removed LNI 192.168.204.123@tcp111 [10166.879108] LNet: Removed LNI 192.168.204.123@tcp112 [10167.903121] LNet: Removed LNI 192.168.204.123@tcp113 [10168.927089] LNet: Removed LNI 192.168.204.123@tcp114 [10168.928196] LNet: Removed LNI 192.168.204.123@tcp115 [10168.929181] LNet: Removed LNI 192.168.204.123@tcp116 [10168.930163] LNet: Removed LNI 192.168.204.123@tcp117 [10168.931116] LNet: Removed LNI 192.168.204.123@tcp118 [10168.932035] LNet: Removed LNI 192.168.204.123@tcp119 [10168.932957] LNet: Removed LNI 192.168.204.123@tcp120 [10168.933893] LNet: Removed LNI 192.168.204.123@tcp121 [10168.934840] LNet: Removed LNI 192.168.204.123@tcp122 [10168.935776] LNet: Removed LNI 192.168.204.123@tcp123 [10168.936749] LNet: Removed LNI 192.168.204.123@tcp124 [10168.937691] LNet: Removed LNI 192.168.204.123@tcp125 [10168.938665] LNet: Removed LNI 192.168.204.123@tcp126 [10168.939611] LNet: Removed LNI 192.168.204.123@tcp127 [10168.940432] LNet: Removed LNI 192.168.204.123@tcp128 [10168.941362] LNet: Removed LNI 192.168.204.123@tcp129 [10168.942324] LNet: Removed LNI 192.168.204.123@tcp130 [10168.943262] LNet: Removed LNI 192.168.204.123@tcp131 [10168.944209] LNet: Removed LNI 192.168.204.123@tcp132 [10168.945180] LNet: Removed LNI 192.168.204.123@tcp133 [10168.946139] LNet: Removed LNI 192.168.204.123@tcp134 [10168.953451] LNet: Removed LNI 192.168.204.123@tcp135 [10169.194898] Key type .llcrypt unregistered [10169.196174] Key type ._llcrypt unregistered [10176.083504] Key type ._llcrypt registered [10176.084370] Key type .llcrypt registered [10176.115614] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [10182.148471] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [10182.427928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10182.435567] alg: No test for adler32 (adler32-zlib) [10183.271248] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [10183.343772] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [10184.919250] Key type lgssc registered [10185.318766] Lustre: Echo OBD driver; http://www.lustre.org/ [10196.286261] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 03:52:27 (1769763147) [10201.783554] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [10202.693243] Key type lgssc unregistered [10202.841548] LNet: 442484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10202.846492] LNetError: 442484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10202.857540] LNet: Removed LNI 192.168.204.123@tcp [10203.110649] Key type .llcrypt unregistered [10203.111828] Key type ._llcrypt unregistered [10209.408551] Key type ._llcrypt registered [10209.409566] Key type .llcrypt registered [10209.439187] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [10209.806714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10209.817584] alg: No test for adler32 (adler32-zlib) [10210.665014] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [10210.738732] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [10212.311192] Key type lgssc registered [10212.652857] Lustre: Echo OBD driver; http://www.lustre.org/ [10215.547465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10215.553360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10216.635419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10216.645919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10216.681298] Lustre: lustre-MDT0000: new disk, initializing [10216.702653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10216.708137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10217.828357] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10221.792174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10221.830811] Lustre: 443654:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10221.855381] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10221.857074] Lustre: Skipped 1 previous similar message [10221.896671] Lustre: lustre-MDT0001: new disk, initializing [10221.930882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10221.941789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10221.945041] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10223.115348] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10226.190604] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10227.375955] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10229.577178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10229.693718] Lustre: lustre-OST0000: new disk, initializing [10229.695725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10229.725416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10230.906487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10230.909500] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10230.956110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10231.550725] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10234.680640] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10235.786743] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing lnet_if_list [10237.324089] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [10237.337206] LNet: Added LNI 192.168.204.123@tcp10 [8/256/0/180] [10239.018144] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [10239.028227] LNet: Added LNI 192.168.204.123@tcp11 [8/256/0/180] [10240.256754] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [10240.267815] LNet: Added LNI 192.168.204.123@tcp12 [8/256/0/180] [10241.581306] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [10241.590900] LNet: Added LNI 192.168.204.123@tcp13 [8/256/0/180] [10242.883982] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [10242.893534] LNet: Added LNI 192.168.204.123@tcp14 [8/256/0/180] [10244.249751] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [10244.259453] LNet: Added LNI 192.168.204.123@tcp15 [8/256/0/180] [10245.587620] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [10245.597430] LNet: Added LNI 192.168.204.123@tcp16 [8/256/0/180] [10246.976935] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [10246.985847] LNet: Added LNI 192.168.204.123@tcp17 [8/256/0/180] [10248.310644] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [10248.321229] LNet: Added LNI 192.168.204.123@tcp18 [8/256/0/180] [10249.724270] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [10249.738441] LNet: Added LNI 192.168.204.123@tcp19 [8/256/0/180] [10251.166071] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [10251.179351] LNet: Added LNI 192.168.204.123@tcp20 [8/256/0/180] [10252.620679] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [10252.632943] LNet: Added LNI 192.168.204.123@tcp21 [8/256/0/180] [10254.121858] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [10254.130758] LNet: Added LNI 192.168.204.123@tcp22 [8/256/0/180] [10255.750330] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [10255.760789] LNet: Added LNI 192.168.204.123@tcp23 [8/256/0/180] [10257.435177] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [10257.452434] LNet: Added LNI 192.168.204.123@tcp24 [8/256/0/180] [10259.039928] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [10259.048960] LNet: Added LNI 192.168.204.123@tcp25 [8/256/0/180] [10260.715920] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [10260.727223] LNet: Added LNI 192.168.204.123@tcp26 [8/256/0/180] [10262.351360] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [10262.360140] LNet: Added LNI 192.168.204.123@tcp27 [8/256/0/180] [10264.039871] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [10264.051600] LNet: Added LNI 192.168.204.123@tcp28 [8/256/0/180] [10265.736072] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [10265.745700] LNet: Added LNI 192.168.204.123@tcp29 [8/256/0/180] [10267.393473] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [10267.402223] LNet: Added LNI 192.168.204.123@tcp30 [8/256/0/180] [10269.123902] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [10269.132870] LNet: Added LNI 192.168.204.123@tcp31 [8/256/0/180] [10270.940238] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [10270.948885] LNet: Added LNI 192.168.204.123@tcp32 [8/256/0/180] [10272.747854] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [10272.757359] LNet: Added LNI 192.168.204.123@tcp33 [8/256/0/180] [10274.678726] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [10274.687949] LNet: Added LNI 192.168.204.123@tcp34 [8/256/0/180] [10276.654833] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [10276.664505] LNet: Added LNI 192.168.204.123@tcp35 [8/256/0/180] [10278.617445] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [10278.628460] LNet: Added LNI 192.168.204.123@tcp36 [8/256/0/180] [10280.635773] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [10280.645940] LNet: Added LNI 192.168.204.123@tcp37 [8/256/0/180] [10282.680200] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [10282.689942] LNet: Added LNI 192.168.204.123@tcp38 [8/256/0/180] [10284.855681] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [10284.867867] LNet: Added LNI 192.168.204.123@tcp39 [8/256/0/180] [10286.981410] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [10286.997871] LNet: Added LNI 192.168.204.123@tcp40 [8/256/0/180] [10289.115681] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [10289.125265] LNet: Added LNI 192.168.204.123@tcp41 [8/256/0/180] [10291.222192] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [10291.232844] LNet: Added LNI 192.168.204.123@tcp42 [8/256/0/180] [10293.439626] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [10293.450086] LNet: Added LNI 192.168.204.123@tcp43 [8/256/0/180] [10295.843801] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [10295.852928] LNet: Added LNI 192.168.204.123@tcp44 [8/256/0/180] [10297.971275] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [10297.979901] LNet: Added LNI 192.168.204.123@tcp45 [8/256/0/180] [10300.165623] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [10300.174814] LNet: Added LNI 192.168.204.123@tcp46 [8/256/0/180] [10302.440643] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [10302.449378] LNet: Added LNI 192.168.204.123@tcp47 [8/256/0/180] [10304.657345] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [10304.666614] LNet: Added LNI 192.168.204.123@tcp48 [8/256/0/180] [10307.032608] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [10307.041196] LNet: Added LNI 192.168.204.123@tcp49 [8/256/0/180] [10309.558745] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [10309.570913] LNet: Added LNI 192.168.204.123@tcp50 [8/256/0/180] [10318.306506] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10318.308120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10318.310976] Lustre: Skipped 1 previous similar message [10322.815260] LustreError: 467674:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10322.845375] Lustre: server umount lustre-MDT0000 complete [10324.025987] LustreError: 444431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769763275 with bad export cookie 12506273670285142358 [10324.029238] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10324.029689] LustreError: 444431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10324.092253] LustreError: 467876:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10324.095510] LustreError: 467876:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10324.139653] Lustre: server umount lustre-MDT0001 complete [10330.923473] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [10333.682310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10344.927105] Lustre: 442928:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769763280/real 1769763280] req@00000000b777a139 x1855731177436288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769763296 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10344.933427] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10344.936557] Lustre: Skipped 1 previous similar message [10345.951097] Lustre: 442929:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769763281/real 1769763281] req@0000000058257ecc x1855731177436416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769763297 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10349.023283] LustreError: 442925:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004424dc63 x1855731177437184/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10349.121471] LustreError: 445389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10349.125630] LustreError: 445389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10349.137667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10350.175086] Lustre: 442926:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769763285/real 1769763285] req@000000006bee1d44 x1855731177436544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769763301 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10350.302890] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10352.886922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10354.138805] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10355.039106] Lustre: 442927:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769763290/real 1769763290] req@000000008fca8c59 x1855731177436800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769763306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10355.044976] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10355.858656] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10356.979929] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10360.287085] Lustre: 442928:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769763295/real 1769763295] req@00000000dd2ad774 x1855731177436928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769763311 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10364.384111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10364.386183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10364.389670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10364.391261] Lustre: Skipped 2 previous similar messages [10365.408898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10370.095192] LustreError: 470364:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10370.098276] LustreError: 470364:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10370.134680] Lustre: server umount lustre-OST0000 complete [10373.599758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10373.602604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10373.606572] Lustre: Skipped 1 previous similar message [10373.608630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10373.610273] Lustre: Skipped 1 previous similar message [10377.402744] LustreError: 470565:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10377.405548] LustreError: 470565:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10377.433880] Lustre: server umount lustre-MDT0000 complete [10378.518091] LustreError: 443645:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769763330 with bad export cookie 12506273670285144360 [10378.520996] LustreError: 443645:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10378.521782] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10378.631894] Lustre: server umount lustre-MDT0001 complete [10381.679306] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [10382.568559] Key type lgssc unregistered [10382.708144] LNet: 471304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10382.710616] LNetError: 471304:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10382.712340] LNet: Removed LNI 192.168.204.123@tcp [10382.713272] LNet: Removed LNI 192.168.204.123@tcp10 [10382.714209] LNet: Removed LNI 192.168.204.123@tcp11 [10382.715139] LNet: Removed LNI 192.168.204.123@tcp12 [10382.716054] LNet: Removed LNI 192.168.204.123@tcp13 [10382.716947] LNet: Removed LNI 192.168.204.123@tcp14 [10382.717845] LNet: Removed LNI 192.168.204.123@tcp15 [10382.718786] LNet: Removed LNI 192.168.204.123@tcp16 [10382.719726] LNet: Removed LNI 192.168.204.123@tcp17 [10382.720668] LNet: Removed LNI 192.168.204.123@tcp18 [10382.721540] LNet: Removed LNI 192.168.204.123@tcp19 [10382.722411] LNet: Removed LNI 192.168.204.123@tcp20 [10382.723368] LNet: Removed LNI 192.168.204.123@tcp21 [10382.724295] LNet: Removed LNI 192.168.204.123@tcp22 [10382.725265] LNet: Removed LNI 192.168.204.123@tcp23 [10382.726218] LNet: Removed LNI 192.168.204.123@tcp24 [10382.727117] LNet: Removed LNI 192.168.204.123@tcp25 [10382.728062] LNet: Removed LNI 192.168.204.123@tcp26 [10382.728927] LNet: Removed LNI 192.168.204.123@tcp27 [10382.729829] LNet: Removed LNI 192.168.204.123@tcp28 [10382.730745] LNet: Removed LNI 192.168.204.123@tcp29 [10382.731691] LNet: Removed LNI 192.168.204.123@tcp30 [10382.732564] LNet: Removed LNI 192.168.204.123@tcp31 [10382.733456] LNet: Removed LNI 192.168.204.123@tcp32 [10382.734377] LNet: Removed LNI 192.168.204.123@tcp33 [10382.735320] LNet: Removed LNI 192.168.204.123@tcp34 [10382.736242] LNet: Removed LNI 192.168.204.123@tcp35 [10382.737166] LNet: Removed LNI 192.168.204.123@tcp36 [10382.738133] LNet: Removed LNI 192.168.204.123@tcp37 [10382.739028] LNet: Removed LNI 192.168.204.123@tcp38 [10382.739893] LNet: Removed LNI 192.168.204.123@tcp39 [10382.740798] LNet: Removed LNI 192.168.204.123@tcp40 [10382.741740] LNet: Removed LNI 192.168.204.123@tcp41 [10382.742645] LNet: Removed LNI 192.168.204.123@tcp42 [10382.743501] LNet: Removed LNI 192.168.204.123@tcp43 [10382.744407] LNet: Removed LNI 192.168.204.123@tcp44 [10382.745305] LNet: Removed LNI 192.168.204.123@tcp45 [10382.746228] LNet: Removed LNI 192.168.204.123@tcp46 [10382.747129] LNet: Removed LNI 192.168.204.123@tcp47 [10382.748047] LNet: Removed LNI 192.168.204.123@tcp48 [10382.748927] LNet: Removed LNI 192.168.204.123@tcp49 [10382.753296] LNet: Removed LNI 192.168.204.123@tcp50 [10382.984699] Key type .llcrypt unregistered [10382.985642] Key type ._llcrypt unregistered [10389.752344] Key type ._llcrypt registered [10389.753227] Key type .llcrypt registered [10389.788749] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [10390.228091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10390.244557] alg: No test for adler32 (adler32-zlib) [10391.083087] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [10391.158882] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [10392.735109] Key type lgssc registered [10393.031145] Lustre: Echo OBD driver; http://www.lustre.org/ [10395.242435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10395.248203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10396.392778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10397.522376] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10400.029072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10400.139815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10401.276109] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10402.901475] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10404.046191] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10406.098149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10406.211309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10407.272116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10407.938181] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10409.563682] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10413.973038] loop0: detected capacity change from 0 to 204800000 [10414.026648] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10414.029798] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10414.035539] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10415.904207] loop0: detected capacity change from 0 to 204800000 [10415.970239] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10415.973165] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10415.976706] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10417.495153] loop0: detected capacity change from 0 to 204800000 [10419.226517] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10419.278567] Lustre: fs15246-OST0000: new disk, initializing [10419.280214] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [10419.295581] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10420.973087] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10424.496057] loop1: detected capacity change from 0 to 204800000 [10426.186367] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10426.219044] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [10426.243478] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [10426.295129] Lustre: fs15246-MDT0000: new disk, initializing [10426.347273] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10426.356461] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10426.402852] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10427.424700] Lustre: 476884:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000005903d75f x1855731366130048/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1769763395 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [10427.482840] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10434.534504] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10436.511193] LustreError: 476884:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [10436.514608] LustreError: 476884:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [10436.517822] LustreError: 476884:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [10439.651498] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10440.314692] LustreError: 478003:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [10440.336935] Lustre: server umount fs15246-MDT0000 complete [10443.089104] LustreError: 478507:0:(obd_class.h:479:obd_check_dev()) Device 29 not setup [10443.090828] LustreError: 478507:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10443.104072] Lustre: server umount fs15246-OST0000 complete [10446.304410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10446.308464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10446.314700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10448.351749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10448.354066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10448.357995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10452.205598] LustreError: 479008:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10452.207826] LustreError: 479008:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [10452.242283] Lustre: server umount lustre-OST0000 complete [10455.008826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10455.009426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10455.013478] Lustre: Skipped 1 previous similar message [10455.014945] Lustre: Skipped 2 previous similar messages [10459.455191] LustreError: 479210:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10459.456719] LustreError: 479210:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10459.481203] Lustre: server umount lustre-MDT0000 complete [10460.127892] LustreError: 473235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10460.131864] LustreError: 473235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10460.624036] LustreError: 472504:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769763412 with bad export cookie 1315510574334354471 [10460.627686] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10460.629568] LustreError: 472504:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10460.739639] Lustre: server umount lustre-MDT0001 complete [10464.044647] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [10464.893841] Key type lgssc unregistered [10465.004353] LNet: 479945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10465.006504] LNetError: 479945:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10465.019273] LNet: Removed LNI 192.168.204.123@tcp [10465.284357] Key type .llcrypt unregistered [10465.285283] Key type ._llcrypt unregistered [10472.372341] Key type ._llcrypt registered [10472.373415] Key type .llcrypt registered [10472.404946] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [10472.709932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10472.773553] alg: No test for adler32 (adler32-zlib) [10473.623842] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [10473.704490] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [10475.279109] Key type lgssc registered [10475.578504] Lustre: Echo OBD driver; http://www.lustre.org/ [10486.834478] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [10489.941201] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [10500.778034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10500.783057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10501.868493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10501.877662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10501.912136] Lustre: lustre-MDT0000: new disk, initializing [10501.935087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10501.940407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10503.104590] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10506.956049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10506.995054] Lustre: 484324:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10507.015783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10507.017317] Lustre: Skipped 1 previous similar message [10507.052345] Lustre: lustre-MDT0001: new disk, initializing [10507.082091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10507.092923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10507.095495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10508.222342] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10511.085958] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10512.154672] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10514.245320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10514.345662] Lustre: lustre-OST0000: new disk, initializing [10514.347452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10514.379550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10515.445689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10515.448275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10515.482131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10516.129981] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10519.000920] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10520.258840] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10520.327954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10521.530628] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10521.599849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10525.669964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10525.671781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10525.672562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10525.679854] Lustre: Skipped 1 previous similar message [10525.684150] LustreError: Skipped 1 previous similar message [10528.303082] LustreError: 487254:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10528.339385] Lustre: server umount lustre-OST0000 complete [10530.787530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10530.788499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10530.799389] Lustre: Skipped 1 previous similar message [10530.803668] Lustre: Skipped 2 previous similar messages [10535.687291] LustreError: 487455:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10535.691211] LustreError: 487455:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10535.730924] Lustre: server umount lustre-MDT0000 complete [10537.003215] LustreError: 484315:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769763488 with bad export cookie 18241312257140204800 [10537.006690] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10537.009706] LustreError: 484315:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10537.079978] LustreError: 487657:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10537.083122] LustreError: 487657:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10537.133332] Lustre: server umount lustre-MDT0001 complete [10544.590868] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [10547.507679] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [10557.205378] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10236 sec ========= 03:58:28 (1769763508) [10557.735981] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:58:29 (1769763509) === [10558.802983] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:58:30 (1769763510) === [10570.867552] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [10571.764060] Key type lgssc unregistered [10571.892611] LNet: 492495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10571.897372] LNetError: 492495:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10571.905513] LNet: Removed LNI 192.168.204.123@tcp [10572.289327] Key type .llcrypt unregistered [10572.290359] Key type ._llcrypt unregistered