[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 499858487 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003070] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008723] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.011139] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.015031] *** VALIDATE selinux *** [ 0.023546] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027635] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028171] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029000] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029128] *** VALIDATE tmpfs *** [ 0.031263] *** VALIDATE proc *** [ 0.032286] *** VALIDATE cgroup *** [ 0.033013] *** VALIDATE cgroup2 *** [ 0.035162] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.040025] Speculative Store Bypass: Vulnerable [ 0.043349] debug: unmapping init [mem 0xffffffff85459000-0xffffffff85460fff] [ 0.045165] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046853] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047033] ... version: 2 [ 0.047839] ... bit width: 48 [ 0.048021] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050021] ... max period: 00007fffffffffff [ 0.051018] ... fixed-purpose events: 3 [ 0.052018] ... event mask: 000000070000000f [ 0.054314] rcu: Hierarchical SRCU implementation. [ 0.056689] smp: Bringing up secondary CPUs ... [ 0.057665] x86: Booting SMP configuration: [ 0.058043] .... node #0, CPUs: #1 #2 #3 [ 0.063018] smp: Brought up 1 node, 4 CPUs [ 0.065020] smpboot: Max logical packages: 1 [ 0.066017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.239024] node 0 deferred pages initialised in 171ms [ 0.243205] devtmpfs: initialized [ 0.244265] x86/mm: Memory block size: 128MB [ 0.248000] gcov: version magic: 0x41383552 [ 0.250320] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.256124] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.260302] pinctrl core: initialized pinctrl subsystem [ 0.262167] [ 0.262857] ************************************************************* [ 0.266015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.269024] ** ** [ 0.272034] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.275014] ** ** [ 0.278016] ** This means that this kernel is built to expose internal ** [ 0.281016] ** IOMMU data structures, which may compromise security on ** [ 0.284018] ** your system. ** [ 0.287017] ** ** [ 0.291015] ** If you see this message and you are not debugging the ** [ 0.294016] ** kernel, report this immediately to your vendor! ** [ 0.297015] ** ** [ 0.301016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.304016] ************************************************************* [ 0.307693] NET: Registered protocol family 16 [ 0.310434] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.314084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.317067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.321021] cpuidle: using governor menu [ 0.323037] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.325486] PCI: Using configuration type 1 for base access [ 0.327147] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.337173] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.338054] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.340072] cryptd: max_cpu_qlen set to 1000 [ 0.343336] ACPI: Added _OSI(Module Device) [ 0.345015] ACPI: Added _OSI(Processor Device) [ 0.347013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.348014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.352491] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.360473] ACPI: Interpreter enabled [ 0.362061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.364009] ACPI: Using IOAPIC for interrupt routing [ 0.366116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.370425] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.379000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.382045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.384020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.388091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.392196] acpiphp: Slot [2] registered [ 0.393141] acpiphp: Slot [3] registered [ 0.395092] acpiphp: Slot [4] registered [ 0.396084] acpiphp: Slot [5] registered [ 0.397109] acpiphp: Slot [6] registered [ 0.399111] acpiphp: Slot [7] registered [ 0.400113] acpiphp: Slot [8] registered [ 0.401111] acpiphp: Slot [9] registered [ 0.402127] acpiphp: Slot [10] registered [ 0.403116] acpiphp: Slot [11] registered [ 0.404000] acpiphp: Slot [12] registered [ 0.404000] acpiphp: Slot [13] registered [ 0.406118] acpiphp: Slot [14] registered [ 0.408089] acpiphp: Slot [15] registered [ 0.409082] acpiphp: Slot [16] registered [ 0.410085] acpiphp: Slot [17] registered [ 0.412102] acpiphp: Slot [18] registered [ 0.413097] acpiphp: Slot [19] registered [ 0.415101] acpiphp: Slot [20] registered [ 0.417101] acpiphp: Slot [21] registered [ 0.418091] acpiphp: Slot [22] registered [ 0.419092] acpiphp: Slot [23] registered [ 0.421121] acpiphp: Slot [24] registered [ 0.422092] acpiphp: Slot [25] registered [ 0.424202] acpiphp: Slot [26] registered [ 0.427125] acpiphp: Slot [27] registered [ 0.428120] acpiphp: Slot [28] registered [ 0.430249] acpiphp: Slot [29] registered [ 0.432119] acpiphp: Slot [30] registered [ 0.433118] acpiphp: Slot [31] registered [ 0.435067] PCI host bridge to bus 0000:00 [ 0.436021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.439027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.441024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.444025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.447027] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.450029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.452177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.456512] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.460399] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.471023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.476554] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.479024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.481017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.484017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.486459] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.489795] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.492045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.495375] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.501017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.514022] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.519883] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.524822] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.534021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.542027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.559023] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.567905] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.575020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.579019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.598026] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.608782] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.614017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.623114] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.638017] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.649369] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.655015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.663034] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.679032] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.690365] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.697015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.701023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.717037] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.725196] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.733016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.740019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.753019] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.763292] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.765347] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.768362] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.770377] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.772238] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.778180] iommu: Default domain type: Passthrough [ 0.779527] SCSI subsystem initialized [ 0.780188] ACPI: bus type USB registered [ 0.781162] usbcore: registered new interface driver usbfs [ 0.784119] usbcore: registered new interface driver hub [ 0.786157] usbcore: registered new device driver usb [ 0.788263] pps_core: LinuxPPS API ver. 1 registered [ 0.790016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.794091] PTP clock support registered [ 0.796195] EDAC MC: Ver: 3.0.0 [ 0.798206] PCI: Using ACPI for IRQ routing [ 0.800885] NetLabel: Initializing [ 0.802016] NetLabel: domain hash size = 128 [ 0.803012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.805091] NetLabel: unlabeled traffic allowed by default [ 0.807151] vgaarb: loaded [ 0.809311] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.811022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.816439] clocksource: Switched to clocksource kvm-clock [ 0.928223] VFS: Disk quotas dquot_6.6.0 [ 0.929500] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.932152] *** VALIDATE ramfs *** [ 0.933357] *** VALIDATE hugetlbfs *** [ 0.934870] pnp: PnP ACPI init [ 0.937484] pnp: PnP ACPI: found 6 devices [ 0.955666] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.959426] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.961671] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.964287] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.966218] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.968890] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.971427] NET: Registered protocol family 2 [ 0.974243] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.979611] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.983360] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.988965] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.992598] TCP: Hash tables configured (established 65536 bind 65536) [ 0.995870] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.999262] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.002252] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.005698] NET: Registered protocol family 1 [ 1.008565] RPC: Registered named UNIX socket transport module. [ 1.010523] RPC: Registered udp transport module. [ 1.012110] RPC: Registered tcp transport module. [ 1.013447] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.015426] NET: Registered protocol family 44 [ 1.016797] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.033391] pci 0000:00:00.0: quirk_natoma+0x0/0x20 took 15318 usecs [ 1.037069] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.040627] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.043290] PCI: CLS 0 bytes, default 64 [ 1.049257] Unpacking initramfs... [ 2.462499] debug: unmapping init [mem 0xffff96627cc54000-0xffff96627ffbffff] [ 2.466680] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.469251] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.472231] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.964416] Initialise system trusted keyrings [ 2.966107] Key type blacklist registered [ 2.969061] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.978581] zbud: loaded [ 2.981573] *** VALIDATE nfs *** [ 2.982734] *** VALIDATE nfs4 *** [ 2.984406] pstore: using deflate compression [ 2.987754] Platform Keyring initialized [ 3.099371] NET: Registered protocol family 38 [ 3.101129] Key type asymmetric registered [ 3.102502] Asymmetric key parser 'x509' registered [ 3.104151] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.109084] io scheduler mq-deadline registered [ 3.110575] io scheduler kyber registered [ 3.111901] io scheduler bfq registered [ 3.113826] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.116485] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.120349] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.122954] ACPI: Power Button [PWRF] [ 3.223600] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.321861] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.517806] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.617312] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.820534] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.851852] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.884058] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.891601] Non-volatile memory driver v1.3 [ 3.893432] Linux agpgart interface v0.103 [ 3.923733] virtio_blk virtio1: [vda] 133856 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.927624] vda: detected capacity change from 0 to 68534272 [ 3.944372] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.947146] vdb: detected capacity change from 0 to 1073741824 [ 3.964395] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.967476] vdc: detected capacity change from 0 to 2621440000 [ 3.988971] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.992143] vdd: detected capacity change from 0 to 2621440000 [ 4.005956] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.014174] vde: detected capacity change from 0 to 4294967296 [ 4.031832] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.035604] vdf: detected capacity change from 0 to 4294967296 [ 4.043530] libphy: Fixed MDIO Bus: probed [ 4.053461] usbcore: registered new interface driver usbserial_generic [ 4.056812] usbserial: USB Serial support registered for generic [ 4.059630] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.065403] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.067485] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.069967] mousedev: PS/2 mouse device common for all mice [ 4.073603] rtc_cmos 00:05: RTC can wake from S4 [ 4.076771] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.080657] rtc_cmos 00:05: registered as rtc0 [ 4.083176] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.083711] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.089270] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.090900] intel_pstate: CPU model not supported [ 4.099790] hid: raw HID events driver (C) Jiri Kosina [ 4.101699] usbcore: registered new interface driver usbhid [ 4.103495] usbhid: USB HID core driver [ 4.105212] drop_monitor: Initializing network drop monitor service [ 4.107401] Initializing XFRM netlink socket [ 4.109392] NET: Registered protocol family 10 [ 4.112448] Segment Routing with IPv6 [ 4.113950] NET: Registered protocol family 17 [ 4.116366] mpls_gso: MPLS GSO support [ 4.123887] RAS: Correctable Errors collector initialized. [ 4.126361] AVX version of gcm_enc/dec engaged. [ 4.128219] AES CTR mode by8 optimization enabled [ 4.207041] sched_clock: Marking stable (4207012525, 0)->(5073213260, -866200735) [ 4.210306] registered taskstats version 1 [ 4.212174] Loading compiled-in X.509 certificates [ 4.214475] zswap: loaded using pool lzo/zbud [ 4.260345] Key type big_key registered [ 4.274034] Key type encrypted registered [ 4.275259] ima: No TPM chip found, activating TPM-bypass! [ 4.277284] ima: Allocated hash algorithm: sha1 [ 4.279307] ima: No architecture policies found [ 4.281072] evm: Initialising EVM extended attributes: [ 4.282928] evm: security.selinux [ 4.284111] evm: security.ima [ 4.285181] evm: security.capability [ 4.286473] evm: HMAC attrs: 0x1 [ 4.289045] rtc_cmos 00:05: setting system clock to 2025-11-18 23:53:49 UTC (1763510029) [ 4.295560] debug: unmapping init [mem 0xffffffff86403000-0xffffffff865fffff] [ 4.298425] debug: unmapping init [mem 0xffffffff85182000-0xffffffff85458fff] [ 4.306412] Write protecting the kernel read-only data: 28672k [ 4.309620] debug: unmapping init [mem 0xffffffff83803000-0xffffffff839fffff] [ 4.312385] debug: unmapping init [mem 0xffffffff84114000-0xffffffff841fffff] [ 4.349492] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.357743] systemd[1]: Detected virtualization kvm. [ 4.359631] systemd[1]: Detected architecture x86-64. [ 4.361610] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.392613] systemd[1]: No hostname configured. [ 4.394347] systemd[1]: Set hostname to . [ 4.396367] random: systemd: uninitialized urandom read (16 bytes read) [ 4.398974] systemd[1]: Initializing machine ID from random generator. [ 4.459175] random: ln: uninitialized urandom read (6 bytes read) [ 4.538642] random: systemd: uninitialized urandom read (16 bytes read) [ 4.541952] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.551463] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 4.555889] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.194345] device-mapper: uevent: version 1.0.3 [ 5.196750] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.917862] virtio_net virtio0 ens2: renamed from eth0 [ 5.922878] random: fast init done [ 5.951236] scsi host0: ata_piix [ 5.985691] scsi host1: ata_piix [ 5.987365] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.990045] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.764446] dracut-initqueue[584]: RTNETLINK answers: File exists [ 10.791341] random: crng init done [ 10.792864] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.220231] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.470832] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.747516] SELinux: Disabled at runtime. [ 12.813849] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.823263] systemd[1]: Detected virtualization kvm. [ 12.825370] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.369860] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.373892] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.379691] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.383701] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.387363] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.397983] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.406532] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ 13.538909] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.925217] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.314611] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.330496] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.430577] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.455611] EDAC sbridge: Ver: 1.1.2 [ 16.321190] Key type dns_resolver registered [ 16.624360] NFS: Registering the id_resolver key type [ 16.626409] Key type id_resolver registered [ 16.627984] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg415-server login: [ 46.038086] libcfs: loading out-of-tree module taints kernel. [ 46.089333] Key type ._llcrypt registered [ 46.091237] Key type .llcrypt registered [ 46.200556] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 58.134245] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 59.256406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 59.268339] alg: No test for adler32 (adler32-zlib) [ 60.446755] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 61.047867] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 62.759199] Key type lgssc registered [ 64.184889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 76.736437] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 77.377947] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 82.517421] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 87.715477] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 94.152750] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.257123] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 117.418245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 117.472837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 117.487905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 118.679608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 118.709040] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 118.772285] Lustre: lustre-MDT0000: new disk, initializing [ 118.859659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 118.883464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 121.899570] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 131.454304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 131.510311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 131.576971] Lustre: 6492:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 131.605621] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 131.611365] Lustre: Skipped 1 previous similar message [ 131.662083] Lustre: lustre-MDT0001: new disk, initializing [ 131.716905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 131.754050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 131.766414] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 134.748125] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 138.725729] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 145.987420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 146.049902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 146.273178] Lustre: lustre-OST0000: new disk, initializing [ 146.280631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 146.335351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 150.563963] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 151.574515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 151.580277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 151.604941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 161.800210] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 161.853096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 161.932324] Lustre: lustre-OST0001: new disk, initializing [ 161.936113] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 161.972510] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 166.445582] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.003687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 171.009812] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 171.064765] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 176.345363] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 180.789363] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 189.938934] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing check_logdir /tmp/testlogs/ [ 193.162162] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing yml_node [ 197.028584] Lustre: DEBUG MARKER: Client: 2.16.61.43 [ 198.785175] hrtimer: interrupt took 7164586 ns [ 199.432426] Lustre: DEBUG MARKER: MDS: 2.16.61.43 [ 201.752415] Lustre: DEBUG MARKER: OSS: 2.16.61.43 [ 203.133642] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Nov 18 18:57:07 EST 2025 [ 220.486117] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 221.855523] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 237.537023] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 237.537660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 237.552774] Lustre: Skipped 3 previous similar messages [ 237.559425] Lustre: Skipped 2 previous similar messages [ 242.662869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 242.669861] Lustre: Skipped 4 previous similar messages [ 242.807622] LustreError: 12195:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 242.909786] Lustre: server umount lustre-MDT0000 complete [ 245.686559] LustreError: 10259:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763510270 with bad export cookie 14854948315094228277 [ 245.693872] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 245.697432] LustreError: 10259:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 247.775989] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 247.778859] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 247.787688] Lustre: Skipped 1 previous similar message [ 247.796038] Lustre: Skipped 1 previous similar message [ 250.786035] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 250.788727] Lustre: Skipped 1 previous similar message [ 251.866144] LustreError: 12397:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 251.869836] LustreError: 12397:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 251.962665] Lustre: server umount lustre-MDT0001 complete [ 254.489465] LustreError: 12598:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 254.491835] LustreError: 12598:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 254.531449] Lustre: server umount lustre-OST0000 complete [ 257.040643] LustreError: 12798:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 257.043773] LustreError: 12798:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 257.116392] Lustre: server umount lustre-OST0001 complete [ 261.246300] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 267.202550] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 274.669670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 280.078499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 284.552876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 289.314067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 293.773870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 293.824603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 294.025180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 294.052477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 294.116046] Lustre: lustre-MDT0000: new disk, initializing [ 294.183800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 294.198548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 296.700555] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 304.963257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 305.028356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 305.099159] Lustre: 14981:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 305.163590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 305.166646] Lustre: Skipped 1 previous similar message [ 305.220791] Lustre: lustre-MDT0001: new disk, initializing [ 305.321205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 305.381255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 305.396358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 308.707440] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 316.586566] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 321.371175] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 328.229706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 328.309202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 328.669698] Lustre: lustre-OST0000: new disk, initializing [ 328.682802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 328.818242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 330.310039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 330.313563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 330.476742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 333.649314] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 339.676280] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 343.131599] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 343.345367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 346.528886] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 346.689315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 351.202614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 351.205616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.209811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 351.217617] Lustre: Skipped 1 previous similar message [ 351.231536] Lustre: Skipped 1 previous similar message [ 354.164215] LustreError: 17921:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 354.175503] LustreError: 17921:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 354.264730] Lustre: server umount lustre-OST0000 complete [ 361.445687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.453988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 361.458663] Lustre: Skipped 1 previous similar message [ 363.322723] LustreError: 18124:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 363.325110] LustreError: 18124:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 363.392351] Lustre: server umount lustre-MDT0000 complete [ 366.518559] LustreError: 14972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763510391 with bad export cookie 14854948315094229810 [ 366.525503] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 366.527800] LustreError: 14972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 366.781214] Lustre: server umount lustre-MDT0001 complete [ 369.900086] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 18:59:54 (1763510394) [ 371.998758] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 373.553890] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 18:59:58 (1763510398) [ 379.346952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 379.698650] LustreError: 18996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 379.717591] LustreError: 18996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 379.781842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 383.219862] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 385.001948] LustreError: 18996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 389.089479] LustreError: 18996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 390.483463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 394.473597] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 399.085992] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 402.086570] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 407.260200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 413.031769] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 419.637318] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 434.150390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 434.154213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 434.157856] Lustre: Skipped 2 previous similar messages [ 438.787088] LustreError: 21261:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 438.792421] LustreError: 21261:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 438.871497] Lustre: server umount lustre-OST0000 complete [ 442.339666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 442.349569] LustreError: Skipped 1 previous similar message [ 442.353380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.159625] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 456.335542] Lustre: server umount lustre-MDT0000 complete [ 459.639208] LustreError: 18977:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763510484 with bad export cookie 14854948315094230692 [ 459.653703] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 459.839101] Lustre: server umount lustre-MDT0001 complete [ 468.112598] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 470.451469] Key type lgssc unregistered [ 470.749871] LNet: 22211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 470.757027] LNetError: 22211:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 470.770734] LNet: Removed LNI 192.168.204.115@tcp [ 471.492170] Key type .llcrypt unregistered [ 471.495450] Key type ._llcrypt unregistered [ 489.367854] Key type ._llcrypt registered [ 489.369767] Key type .llcrypt registered [ 489.455197] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 490.071787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 490.157634] alg: No test for adler32 (adler32-zlib) [ 491.177126] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 491.338489] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 492.975188] Key type lgssc registered [ 493.988803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 501.148807] loop: module loaded [ 501.271581] loop0: detected capacity change from 0 to 204800000 [ 501.499430] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 501.510954] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 501.837297] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 501.903605] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 507.377717] loop0: detected capacity change from 0 to 204800000 [ 507.583187] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 507.596390] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 507.604292] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 507.753828] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 512.480663] loop0: detected capacity change from 0 to 204800000 [ 517.195477] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 517.257554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 517.280707] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.520644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 518.579928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 518.664719] Lustre: lustre-MDT0000: new disk, initializing [ 518.725630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 518.753383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 522.162843] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 529.762917] loop1: detected capacity change from 0 to 204800000 [ 534.498743] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 534.529915] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 534.714802] Lustre: lustre-OST07c6: new disk, initializing [ 534.719859] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 534.723355] Lustre: Skipped 1 previous similar message [ 534.791589] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 535.860610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 535.871202] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 535.896795] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 538.915199] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 542.330924] Lustre: 24351:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 542.337373] Lustre: 24351:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 542.346898] Lustre: 24351:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 542.352060] Lustre: 24351:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 542.360377] Lustre: 24351:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 542.367044] Lustre: 24351:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 555.509233] Lustre: 24352:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 555.513570] Lustre: 24352:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 555.517352] Lustre: 24352:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 555.521780] Lustre: 24352:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 555.530667] Lustre: 24352:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 555.534261] Lustre: 24352:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 559.864650] Lustre: 24352:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 559.869427] Lustre: 24352:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 559.873929] Lustre: 24352:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 559.878092] Lustre: 24352:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 559.882286] Lustre: 24352:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 559.886316] Lustre: 24352:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 559.890219] Lustre: 24352:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 559.895264] Lustre: 24352:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 559.899986] Lustre: 24352:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 559.904714] Lustre: 24352:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 559.909494] Lustre: 24352:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 559.917320] Lustre: 24352:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 563.100512] Lustre: 24353:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 563.104962] Lustre: 24353:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 563.111223] Lustre: 24353:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 563.114254] Lustre: 24353:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 563.128307] Lustre: 24353:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 563.132137] Lustre: 24353:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 563.135966] Lustre: 24353:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 563.140716] Lustre: 24353:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 563.144248] Lustre: 24353:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 563.147019] Lustre: 24353:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 563.150617] Lustre: 24353:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 563.155324] Lustre: 24353:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 567.357454] Lustre: 24352:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 567.365563] Lustre: 24352:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 567.373941] Lustre: 24352:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 567.383129] Lustre: 24352:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 567.393216] Lustre: 24352:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 567.399685] Lustre: 24352:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 567.404245] Lustre: 24352:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 567.409267] Lustre: 24352:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 567.419271] Lustre: 24352:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 567.424778] Lustre: 24352:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 567.430420] Lustre: 24352:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 567.434592] Lustre: 24352:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 578.185992] Lustre: 24353:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 578.211946] Lustre: 24353:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 5 previous similar messages [ 578.227531] Lustre: 24353:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 578.238260] Lustre: 24353:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 578.256037] Lustre: 24353:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 578.277056] Lustre: 24353:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 578.296293] Lustre: 24353:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 578.317314] Lustre: 24353:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 578.341040] Lustre: 24353:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 578.344744] Lustre: 24353:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 578.363258] Lustre: 24353:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 578.366351] Lustre: 24353:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 628.194156] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 628.195105] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.214227] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 630.966341] LustreError: 26536:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 631.148653] Lustre: server umount lustre-OST07c6 complete [ 639.714138] LustreError: 27037:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 639.717318] LustreError: 27037:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 639.918435] Lustre: server umount lustre-MDT0000 complete [ 652.669876] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 655.386635] Key type lgssc unregistered [ 655.630316] LNet: 27871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.642246] LNetError: 27871:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.659986] LNet: Removed LNI 192.168.204.115@tcp [ 656.472084] Key type .llcrypt unregistered [ 656.473787] Key type ._llcrypt unregistered [ 671.592811] Key type ._llcrypt registered [ 671.594694] Key type .llcrypt registered [ 671.684370] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 672.543907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 672.602332] alg: No test for adler32 (adler32-zlib) [ 673.597117] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 673.781766] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 675.409276] Key type lgssc registered [ 676.318236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 682.343594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 682.360607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 683.645948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 686.821736] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 693.226814] LustreError: 29100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 693.238114] LustreError: 29100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 693.784186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 694.018747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 696.934717] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 701.169464] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 704.297302] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 710.537728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 710.820925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.369242] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 720.197287] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 728.239491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 743.918772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 743.920810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 743.922080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 743.953575] Lustre: Skipped 1 previous similar message [ 748.677831] LustreError: 31573:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 748.769920] Lustre: server umount lustre-OST0000 complete [ 754.146830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 754.150034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 754.178219] Lustre: Skipped 1 previous similar message [ 754.204495] Lustre: Skipped 2 previous similar messages [ 757.884166] LustreError: 31775:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 757.887201] LustreError: 31775:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 757.958729] Lustre: server umount lustre-MDT0000 complete [ 759.264204] LustreError: 29100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 759.302406] LustreError: 29100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 761.088732] LustreError: 29082:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763510786 with bad export cookie 757850540326002245 [ 761.094957] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 761.104063] LustreError: 29082:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 761.305529] LustreError: 31976:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 761.335775] LustreError: 31976:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 761.592467] Lustre: server umount lustre-MDT0001 complete [ 770.006424] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 772.723527] Key type lgssc unregistered [ 773.053883] LNet: 32507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.069283] LNetError: 32507:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 773.084703] LNet: Removed LNI 192.168.204.115@tcp [ 773.947767] Key type .llcrypt unregistered [ 773.949943] Key type ._llcrypt unregistered [ 789.841729] Key type ._llcrypt registered [ 789.843178] Key type .llcrypt registered [ 789.965814] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 790.917381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 791.031962] alg: No test for adler32 (adler32-zlib) [ 792.277782] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 792.531391] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 794.184226] Key type lgssc registered [ 795.313808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 802.319518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 802.345733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 803.783948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 807.449714] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 813.028206] LustreError: 33751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 813.048349] LustreError: 33751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 814.333381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 814.607980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 817.610055] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 821.884544] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 824.881176] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 831.195172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 831.528434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 837.602482] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 842.973591] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 847.973170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 858.088305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 858.089365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 858.099837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 858.113253] Lustre: Skipped 1 previous similar message [ 858.122637] Lustre: Skipped 1 previous similar message [ 863.201104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 863.204141] Lustre: Skipped 1 previous similar message [ 863.371328] LustreError: 36019:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 863.483466] Lustre: server umount lustre-OST0000 complete [ 868.322179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 868.326626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 868.327943] Lustre: Skipped 1 previous similar message [ 868.332372] Lustre: Skipped 1 previous similar message [ 872.803457] LustreError: 36222:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 872.809611] LustreError: 36222:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 872.916887] Lustre: server umount lustre-MDT0000 complete [ 873.443647] LustreError: 34822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 873.453683] LustreError: 34822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 876.548092] LustreError: 34469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763510901 with bad export cookie 3445295135828852478 [ 876.555084] LustreError: 34469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 876.556162] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 876.697249] LustreError: 36423:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 876.700294] LustreError: 36423:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 876.957345] Lustre: server umount lustre-MDT0001 complete [ 885.168949] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 887.657021] Key type lgssc unregistered [ 887.920354] LNet: 36956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.927553] LNetError: 36956:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 887.948696] LNet: Removed LNI 192.168.204.115@tcp [ 888.599398] Key type .llcrypt unregistered [ 888.600855] Key type ._llcrypt unregistered [ 905.631699] Key type ._llcrypt registered [ 905.635474] Key type .llcrypt registered [ 905.742570] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 906.741364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 906.752615] alg: No test for adler32 (adler32-zlib) [ 907.853211] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 908.094268] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 909.735432] Key type lgssc registered [ 910.749555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 917.943090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 917.972795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 919.649291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 923.319994] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 929.761798] LustreError: 38338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 929.778159] LustreError: 38338:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 931.143766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 931.584475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 935.144963] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 939.475914] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 942.435876] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 946.884777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 947.134450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 951.879617] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 956.006621] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 964.528175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 971.971846] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 972.768556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 972.783951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 972.807141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 972.988757] LustreError: 40605:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 973.085312] Lustre: server umount lustre-MDT0000 complete [ 974.821483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 974.823823] LustreError: 38339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 974.918982] LustreError: 38339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 976.688454] LustreError: 38319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763511001 with bad export cookie 1034493812405372597 [ 976.693385] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 976.722437] LustreError: 38319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 976.843892] LustreError: 40809:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 976.846592] LustreError: 40809:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 977.045996] Lustre: server umount lustre-MDT0001 complete [ 989.048453] LustreError: 38318:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.15@tcp arrived at 1763511014 with bad export cookie 1034493812405373227 [ 989.058494] LustreError: 38318:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1007.295922] LustreError: 41031:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1007.299656] LustreError: 41031:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1007.504239] Lustre: server umount lustre-OST0000 complete [ 1019.078389] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1022.095483] Key type lgssc unregistered [ 1022.358499] LNet: 41870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.366398] LNetError: 41870:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.379810] LNet: Removed LNI 192.168.204.115@tcp [ 1023.093519] Key type .llcrypt unregistered [ 1023.095807] Key type ._llcrypt unregistered [ 1045.556450] Key type ._llcrypt registered [ 1045.558670] Key type .llcrypt registered [ 1045.705763] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1046.760723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1046.836035] alg: No test for adler32 (adler32-zlib) [ 1048.035944] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 1048.261867] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1049.927518] Key type lgssc registered [ 1051.030714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1058.643661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1058.664789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1060.038978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1063.378550] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1070.260872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1070.568097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1074.088504] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1079.640244] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1083.837526] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1090.687843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1091.022089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1095.926134] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1100.370484] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1106.425783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1116.354396] Lustre: lustre-OST0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 1116.640078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1116.647919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.678471] Lustre: Skipped 1 previous similar message [ 1121.091221] LustreError: 45535:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1121.208650] Lustre: server umount lustre-OST0000 complete [ 1141.219105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.221129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1141.254778] Lustre: Skipped 1 previous similar message [ 1141.274167] Lustre: Skipped 3 previous similar messages [ 1143.509643] LustreError: 45910:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1143.512719] LustreError: 45910:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1143.628372] Lustre: server umount lustre-MDT0000 complete [ 1146.345719] LustreError: 45226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1146.361911] LustreError: 45226:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1146.941584] LustreError: 43249:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763511172 with bad export cookie 15041456815358593872 [ 1146.947816] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1146.950169] LustreError: 43249:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1153.112672] LustreError: 46114:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1153.115219] LustreError: 46114:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1153.255397] Lustre: server umount lustre-MDT0001 complete [ 1162.471983] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1164.928848] Key type lgssc unregistered [ 1165.186594] LNet: 46647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.192101] LNetError: 46647:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1165.205561] LNet: Removed LNI 192.168.204.115@tcp [ 1166.011240] Key type .llcrypt unregistered [ 1166.016784] Key type ._llcrypt unregistered [ 1184.261723] Key type ._llcrypt registered [ 1184.263824] Key type .llcrypt registered [ 1184.377845] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1185.260121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1185.314116] alg: No test for adler32 (adler32-zlib) [ 1186.418903] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 1186.601698] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1188.256175] Key type lgssc registered [ 1189.347330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1196.665416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1196.701516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.422733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1202.011350] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1208.804320] LustreError: 48024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1208.818621] LustreError: 48024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1209.940332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1210.226037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1213.847616] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1218.685637] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1222.587110] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1228.851100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1229.159607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1234.171632] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1239.188639] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1244.529845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1248.583669] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1251.557904] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1263.690953] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-11-1819h14m47s [ 1265.315542] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1268.192807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.200676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1268.203996] Lustre: Skipped 2 previous similar messages [ 1268.220825] Lustre: Skipped 2 previous similar messages [ 1269.285752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1270.979211] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 1270.984749] Lustre: Skipped 2 previous similar messages [ 1273.345634] LustreError: 50588:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1273.355806] LustreError: 50588:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (000000002ba67ff7) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1273.551095] Lustre: server umount lustre-MDT0000 complete [ 1275.375845] LustreError: 48020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1275.396755] LustreError: 48020:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1276.596636] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1277.102842] LustreError: 48020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.515348] LustreError: 48736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.531114] LustreError: 48736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1281.785584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.865425] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1282.136742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.209707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1283.124101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1285.657994] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1289.168275] Lustre: DEBUG MARKER: Wait for df (36074) ... [ 1291.001410] Lustre: DEBUG MARKER: done [ 1296.864715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1296.872952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1296.891810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1296.900595] Lustre: Skipped 1 previous similar message [ 1297.378440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1301.040179] LustreError: 51647:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1301.065792] LustreError: 51647:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1301.263267] Lustre: server umount lustre-OST0000 complete [ 1308.640966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1308.644840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1308.649368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1308.651441] Lustre: Skipped 2 previous similar messages [ 1308.662763] Lustre: Skipped 3 previous similar messages [ 1319.392196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1319.487397] LustreError: 51849:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1319.490196] LustreError: 51849:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1319.565420] Lustre: server umount lustre-MDT0000 complete [ 1321.956187] LustreError: 48024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1321.992665] LustreError: 48024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1322.804690] LustreError: 48005:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763511348 with bad export cookie 2782656566056209527 [ 1322.814552] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1322.820622] LustreError: 48005:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1322.909966] LustreError: 52052:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1322.913173] LustreError: 52052:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1323.063228] Lustre: server umount lustre-MDT0001 complete [ 1331.728612] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1333.999950] Key type lgssc unregistered [ 1334.232343] LNet: 52584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1334.245915] LNetError: 52584:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1334.257806] LNet: Removed LNI 192.168.204.115@tcp [ 1335.069544] Key type .llcrypt unregistered [ 1335.071877] Key type ._llcrypt unregistered [ 1362.608291] Key type ._llcrypt registered [ 1362.610116] Key type .llcrypt registered [ 1362.712968] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 1375.897386] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1377.128305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1377.176936] alg: No test for adler32 (adler32-zlib) [ 1378.294768] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 1378.535464] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1380.167175] Key type lgssc registered [ 1381.037979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1387.880330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1393.839235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1398.877732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1405.314395] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1411.746952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1411.807759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1411.821351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1413.024518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1413.062985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1413.151529] Lustre: lustre-MDT0000: new disk, initializing [ 1413.236799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1413.261956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1418.588775] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1432.144794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1432.202324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1432.256894] Lustre: 56536:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1432.300075] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1432.306326] Lustre: Skipped 1 previous similar message [ 1432.374309] Lustre: lustre-MDT0001: new disk, initializing [ 1432.478447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1432.510319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1432.514931] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1436.593490] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1443.927765] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1447.969668] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1456.627981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1456.720765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1457.001436] Lustre: lustre-OST0000: new disk, initializing [ 1457.004792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1457.108066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1462.760472] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1466.455511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1466.464660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1466.657448] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1469.836720] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1473.573833] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1473.761855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1476.717803] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1476.948501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1482.211269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1482.211287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1482.214623] LustreError: Skipped 1 previous similar message [ 1482.216762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1482.234303] Lustre: Skipped 1 previous similar message [ 1484.533754] LustreError: 59474:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1484.640501] Lustre: server umount lustre-OST0000 complete [ 1488.866931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1488.876536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1488.883435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1488.886065] Lustre: Skipped 1 previous similar message [ 1492.448129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1492.455404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1494.119069] LustreError: 59676:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1494.122333] LustreError: 59676:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1494.196691] Lustre: server umount lustre-MDT0000 complete [ 1497.396598] LustreError: 56528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763511522 with bad export cookie 13570186583559708897 [ 1497.403220] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1497.419328] LustreError: 56528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1497.595068] LustreError: 59879:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1497.612156] LustreError: 59879:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1497.788847] Lustre: server umount lustre-MDT0001 complete [ 1505.389900] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 19:18:50 (1763511530) [ 1510.934588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1511.500882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1515.375285] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1516.514489] LustreError: 60548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1516.536989] LustreError: 60548:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1521.635091] LustreError: 60548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1522.537940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1522.953679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1526.985709] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1533.419793] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1537.492948] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1543.987133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1544.363729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1549.385732] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1554.245389] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1564.588669] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-11-1819h19m48s [ 1565.865104] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1572.006850] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1574.120870] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1574.181361] LustreError: 60545:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1594.572046] Lustre: lustre-MDT0000: Client bb081363-49dc-470d-aaa6-604ec6b8fbe0 (at 192.168.204.15@tcp) reconnecting [ 1614.199387] LustreError: 60545:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1616.412305] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1618.828574] Lustre: DEBUG MARKER: done [ 1625.575391] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1625.588273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1625.614544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1625.620172] Lustre: Skipped 1 previous similar message [ 1628.645051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1629.688108] LustreError: 63551:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1629.690885] LustreError: 63551:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1629.859963] Lustre: server umount lustre-OST0000 complete [ 1633.762057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1633.767703] Lustre: Skipped 1 previous similar message [ 1633.774689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1633.777709] Lustre: Skipped 3 previous similar messages [ 1639.466076] LustreError: 63755:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1639.484251] LustreError: 63755:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1639.581948] Lustre: server umount lustre-MDT0000 complete [ 1639.842776] LustreError: 61566:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1639.860844] LustreError: 61566:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1643.791734] LustreError: 60528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763511669 with bad export cookie 13570186583559709779 [ 1643.797603] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1644.368755] Lustre: server umount lustre-MDT0001 complete [ 1655.094638] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1657.666491] Key type lgssc unregistered [ 1657.923855] LNet: 64490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1657.928221] LNetError: 64490:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.982316] LNet: Removed LNI 192.168.204.115@tcp [ 1659.784688] Key type .llcrypt unregistered [ 1659.786659] Key type ._llcrypt unregistered [ 1691.714384] Key type ._llcrypt registered [ 1691.716152] Key type .llcrypt registered [ 1691.824728] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 1707.197957] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1708.036161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1708.052687] alg: No test for adler32 (adler32-zlib) [ 1709.096645] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 1709.322304] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1710.992577] Key type lgssc registered [ 1712.076370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1720.250050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1727.271983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1734.348319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1741.903594] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1749.008870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1749.066842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1749.087273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1750.325583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1750.386797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1750.461348] Lustre: lustre-MDT0000: new disk, initializing [ 1750.533055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1750.560628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1754.901382] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1766.899381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1767.014641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1767.144259] Lustre: 68441:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1767.196600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1767.199938] Lustre: Skipped 1 previous similar message [ 1767.252565] Lustre: lustre-MDT0001: new disk, initializing [ 1767.345781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1767.404893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1767.418987] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1771.812286] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1779.204549] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1782.441502] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1788.636738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1788.769942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1789.002776] Lustre: lustre-OST0000: new disk, initializing [ 1789.007330] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1789.092516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1794.974683] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1795.178866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1795.195885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1795.393290] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1802.175840] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1805.685541] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1805.860815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1809.211837] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1809.387717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1810.917484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1810.923509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.938618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1810.943593] Lustre: Skipped 1 previous similar message [ 1816.039786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1816.046875] Lustre: Skipped 1 previous similar message [ 1816.775126] LustreError: 71380:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1816.861293] Lustre: server umount lustre-OST0000 complete [ 1821.157891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1821.161953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1821.177443] Lustre: Skipped 2 previous similar messages [ 1825.947785] LustreError: 71582:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1825.950860] LustreError: 71582:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1826.050768] Lustre: server umount lustre-MDT0000 complete [ 1829.358421] LustreError: 68434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763511854 with bad export cookie 17015914242542687955 [ 1829.365792] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1829.371696] LustreError: 68434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1829.497383] LustreError: 71783:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1829.507194] LustreError: 71783:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1829.661609] Lustre: server umount lustre-MDT0001 complete [ 1836.215743] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 19:24:20 (1763511860) [ 1837.514306] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1839.365734] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 19:24:23 (1763511863) [ 1851.039755] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1854.855193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1855.206432] LustreError: 72831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1855.226851] LustreError: 72831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1855.259894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1856.181505] Lustre: Failing over lustre-MDT0000 [ 1856.194085] LustreError: 72859:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1856.216426] LustreError: 72859:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1856.222410] LustreError: 72859:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1856.330026] LustreError: 72911:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1856.333838] LustreError: 72911:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1856.518241] Lustre: server umount lustre-MDT0000 complete [ 1862.393867] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 19:24:47 (1763511887) [ 1868.111691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1868.497152] LustreError: 73478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1868.547260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1871.883212] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1873.891416] LustreError: 73478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1877.986841] LustreError: 73478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1879.975850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1880.379087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1885.117135] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1891.247622] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1895.233797] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1902.068482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1902.373961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1907.152490] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1913.241965] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1922.494950] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1929.190577] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.191623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1929.203375] Lustre: Skipped 2 previous similar messages [ 1929.212811] Lustre: Skipped 3 previous similar messages [ 1929.565704] LustreError: 75821:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1929.570753] LustreError: 75821:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1929.725728] Lustre: server umount lustre-MDT0000 complete [ 1932.939205] LustreError: 73460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763511958 with bad export cookie 17015914242542689110 [ 1932.942289] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1932.946792] LustreError: 73460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1933.204364] Lustre: server umount lustre-MDT0001 complete [ 1936.203919] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1939.732678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1947.246518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1950.628591] Lustre: 66652:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763511959/real 1763511959] req@000000009346e46e x1849176080061440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763511975 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1950.658831] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.698656] Lustre: 66650:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763511962/real 1763511962] req@0000000014cab48c x1849176080061568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763511978 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1955.487122] Lustre: 66651:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763511964/real 1763511964] req@000000004fb0a7da x1849176080061696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763511980 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1957.859267] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xec24abd9734462c6 [ 1957.868519] Lustre: MGC192.168.204.115@tcp: Connection restored to 0@lo (at 0@lo) [ 1958.119837] LustreError: 75085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1958.201024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1958.295502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1959.210750] Lustre: 66652:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763511968/real 1763511968] req@000000007f1a117f x1849176080061952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763511984 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1961.330414] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1964.319278] Lustre: 66653:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763511973/real 1763511973] req@000000001e6a7e99 x1849176080062080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763511989 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1968.319419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1969.636545] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1971.381238] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1975.330808] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1978.052539] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1983.970751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1983.973614] LustreError: Skipped 1 previous similar message [ 1983.995967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1984.022418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1989.149380] LustreError: 78287:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1989.152059] LustreError: 78287:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1989.288230] Lustre: server umount lustre-MDT0000 complete [ 1993.185874] LustreError: 77605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1993.206623] LustreError: 77605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 1993.491445] LustreError: 73458:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763512018 with bad export cookie 17015914242542691014 [ 1993.495244] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1993.513526] LustreError: 73458:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1994.069889] Lustre: server umount lustre-MDT0001 complete [ 2000.880825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2009.426400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2014.687350] Lustre: 66650:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763512023/real 1763512023] req@000000004dae5654 x1849176080086272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763512039 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2014.702709] Lustre: 66650:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2014.706039] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.711673] Lustre: Skipped 2 previous similar messages [ 2017.826325] LustreError: 66649:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000072046e2 x1849176080087040/t0(0) o250->MGC192.168.204.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2018.122174] LustreError: 77605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.228628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2020.721425] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2023.272750] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2023.278704] Lustre: Skipped 1 previous similar message [ 2027.545511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2027.839717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2027.846111] Lustre: Skipped 2 previous similar messages [ 2031.734163] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2033.133083] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2037.434246] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2040.776520] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2048.483446] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2048.488225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2048.493071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2048.541214] Lustre: Skipped 2 previous similar messages [ 2048.569062] Lustre: Skipped 8 previous similar messages [ 2051.780838] LustreError: 80746:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2051.782911] LustreError: 80746:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2051.870932] Lustre: server umount lustre-MDT0000 complete [ 2053.608139] LustreError: 79347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2053.618740] LustreError: 79347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2055.497147] LustreError: 75823:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763512080 with bad export cookie 17015914242542692386 [ 2055.500231] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2055.531023] LustreError: 75823:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2055.846689] Lustre: server umount lustre-MDT0001 complete [ 2062.137908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2065.494921] Lustre: DEBUG MARKER: files compared the same [ 2077.537769] Lustre: server umount lustre-OST0000 complete [ 2090.960671] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2093.570524] Key type lgssc unregistered [ 2093.797200] LNet: 82380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2093.804899] LNetError: 82380:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2093.824054] LNet: Removed LNI 192.168.204.115@tcp [ 2094.598610] Key type .llcrypt unregistered [ 2094.600585] Key type ._llcrypt unregistered [ 2111.799371] Key type ._llcrypt registered [ 2111.801305] Key type .llcrypt registered [ 2111.878696] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2112.599921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2112.627661] alg: No test for adler32 (adler32-zlib) [ 2113.671939] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 2113.831456] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2115.519229] Key type lgssc registered [ 2116.257345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2122.493601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2122.513675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.893658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2127.652817] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug +malloc all [ 2134.506165] LustreError: 83757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.534416] LustreError: 83757:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2135.100073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2135.424879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2140.580281] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug +malloc all [ 2146.526600] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2150.852589] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2157.733406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2158.131458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.375769] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug +malloc all [ 2168.216461] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2169.385917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2179.555057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.555849] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2179.564711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2184.672973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2184.675080] Lustre: Skipped 1 previous similar message [ 2185.030570] LustreError: 86001:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2185.119608] Lustre: server umount lustre-OST0000 complete [ 2189.792317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.797595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2189.799876] Lustre: Skipped 2 previous similar messages [ 2189.804574] Lustre: Skipped 1 previous similar message [ 2194.254488] LustreError: 86203:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2194.262628] LustreError: 86203:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2194.342950] Lustre: server umount lustre-MDT0000 complete [ 2194.911745] LustreError: 83757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.934867] LustreError: 83757:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2197.874140] LustreError: 83737:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763512223 with bad export cookie 2414855377501541849 [ 2197.875380] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2197.882617] LustreError: 83737:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2198.069406] LustreError: 86404:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2198.084826] LustreError: 86404:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2198.274435] Lustre: server umount lustre-MDT0001 complete [ 2207.205533] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2209.977886] Key type lgssc unregistered [ 2210.260696] LNet: 86938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2210.268959] LNetError: 86938:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2210.291578] LNet: Removed LNI 192.168.204.115@tcp [ 2211.126807] Key type .llcrypt unregistered [ 2211.128627] Key type ._llcrypt unregistered [ 2228.192377] Key type ._llcrypt registered [ 2228.193909] Key type .llcrypt registered [ 2228.270189] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2228.957240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2229.014277] alg: No test for adler32 (adler32-zlib) [ 2230.112480] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 2230.318823] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2231.975200] Key type lgssc registered [ 2233.056780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2239.396699] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2239.425176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2254.947035] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2260.063245] LustreError: 88287:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [ 2286.752035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2290.879541] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2294.645432] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2299.838985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2299.970276] Lustre: *** cfs_fail_loc=706, val=0*** [ 2300.262801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2300.312642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2302.661964] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2306.533142] LustreError: 89259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2306.541424] LustreError: 89259:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2308.080915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2308.334375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2310.908512] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2314.657224] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2317.460212] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2320.864509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2320.874334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2320.893492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2323.940196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2323.942925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2323.955338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2325.365639] LustreError: 90614:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2325.452972] Lustre: server umount lustre-OST0000 complete [ 2329.057057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2329.060845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2329.070703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2329.073711] Lustre: Skipped 2 previous similar messages [ 2334.177931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2334.189665] Lustre: Skipped 1 previous similar message [ 2339.297063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2339.306781] Lustre: Skipped 1 previous similar message [ 2342.367504] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2342.434586] LustreError: 90816:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2342.439917] LustreError: 90816:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2342.503382] Lustre: server umount lustre-MDT0000 complete [ 2344.417541] LustreError: 89264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2344.425702] LustreError: 89264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2345.395430] LustreError: 89990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763512370 with bad export cookie 643185265083624740 [ 2345.399201] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2345.404024] LustreError: 89990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2345.514664] LustreError: 91020:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2345.519641] LustreError: 91020:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2345.704772] Lustre: server umount lustre-MDT0001 complete [ 2354.246943] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2356.879451] Key type lgssc unregistered [ 2357.145374] LNet: 91551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2357.161612] LNetError: 91551:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2357.197180] LNet: Removed LNI 192.168.204.115@tcp [ 2358.006759] Key type .llcrypt unregistered [ 2358.014956] Key type ._llcrypt unregistered [ 2378.973964] Key type ._llcrypt registered [ 2378.976960] Key type .llcrypt registered [ 2379.102502] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2380.194207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2380.254524] alg: No test for adler32 (adler32-zlib) [ 2381.481819] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 2381.813760] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2383.519210] Key type lgssc registered [ 2384.958277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2390.729906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2390.756609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2394.313796] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2399.856327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2400.233682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2403.018881] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2405.346210] LustreError: 93610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2405.355421] LustreError: 93610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2408.726575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2409.001603] LustreError: 93610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.011675] LustreError: 93610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2409.027576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2410.019801] LustreError: 93614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2410.031679] LustreError: 93614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2412.890445] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2415.074151] LustreError: 93614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.084234] LustreError: 93614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2418.603132] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2418.611212] Lustre: MGS: Client 17109980-6de7-473a-92ce-b059ba60c564 (at 0@lo) reconnecting [ 2418.803804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2418.811702] LustreError: 93625:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2418.888691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2421.298168] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2434.018766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2434.024791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.040971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2434.534940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.542120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2438.207155] LustreError: 95574:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2438.297448] Lustre: server umount lustre-OST0000 complete [ 2444.768347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2444.769455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2444.781617] Lustre: Skipped 1 previous similar message [ 2444.786161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2444.795444] Lustre: Skipped 1 previous similar message [ 2447.141544] LustreError: 95775:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2447.152628] LustreError: 95775:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2447.175971] Lustre: server umount lustre-MDT0000 complete [ 2449.894491] LustreError: 95125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.919485] LustreError: 95125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2450.316651] LustreError: 95976:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2450.319819] LustreError: 95976:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2450.473981] Lustre: server umount lustre-MDT0001 complete [ 2453.831147] LustreError: 96177:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2453.834182] LustreError: 96177:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2453.872324] Lustre: server umount MGS complete [ 2461.622824] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 19:34:46 (1763512486) [ 2486.648816] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 2492.897695] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2500.549954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2507.470947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2514.198232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2519.862922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2525.341263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2525.398958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2528.081703] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2535.671810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2535.714984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2535.930341] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2535.963184] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2536.018676] Lustre: lustre-MDT0001: new disk, initializing [ 2536.082378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2536.112352] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2536.128877] Lustre: 100701:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2536.138208] LustreError: 100701:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2538.845536] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2547.047529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2547.099568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2547.257390] Lustre: lustre-OST0000: new disk, initializing [ 2547.260583] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2547.320360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.512834] Lustre: 101472:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000004925e3ca x1849176785719936/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763512589 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2551.164139] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2556.672930] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2561.447872] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2561.473451] Lustre: MGS: Client 063b80dd-5a17-41d3-9df9-5e98c1e57491 (at 0@lo) reconnecting [ 2561.514799] Lustre: 99979:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2561.537814] LustreError: 99979:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2561.573949] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2561.631855] Lustre: lustre-MDT0000: new disk, initializing [ 2561.736826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2561.759554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2561.819644] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2562.784115] Lustre: 102415:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000007e96f1c3 x1849176785726080/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1763512603 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2565.145947] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2569.185686] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2570.938697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2570.947724] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2571.940185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2571.950705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2571.975170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2592.740603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2592.752979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.781900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2598.657109] LustreError: 102936:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2598.664539] LustreError: 102936:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2598.841638] Lustre: server umount lustre-OST0000 complete [ 2605.540629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2605.546526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2605.554651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2605.554660] Lustre: Skipped 2 previous similar messages [ 2605.567793] Lustre: Skipped 2 previous similar messages [ 2615.779248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2615.782044] Lustre: Skipped 4 previous similar messages [ 2616.287211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2616.379420] LustreError: 103139:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2616.384610] LustreError: 103139:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2616.421832] Lustre: server umount lustre-MDT0000 complete [ 2633.183357] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2633.268983] LustreError: 103344:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 2633.276645] LustreError: 103344:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2633.407584] Lustre: server umount lustre-MDT0001 complete [ 2636.424422] Lustre: server umount MGS complete [ 2645.530777] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 19:37:50 (1763512670) [ 2660.476990] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2662.710878] Key type lgssc unregistered [ 2662.948740] LNet: 104882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2662.963438] LNetError: 104882:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2662.987714] LNet: Removed LNI 192.168.204.115@tcp [ 2663.751255] Key type .llcrypt unregistered [ 2663.753078] Key type ._llcrypt unregistered [ 2676.698461] Key type ._llcrypt registered [ 2676.708339] Key type .llcrypt registered [ 2676.808516] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2677.580001] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2677.602370] alg: No test for adler32 (adler32-zlib) [ 2678.606110] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 2678.847625] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2680.519241] Key type lgssc registered [ 2681.503813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2689.207155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2689.236924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.592777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2694.003185] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2700.768170] LustreError: 106065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.792108] LustreError: 106065:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2701.780556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2702.138618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2705.413710] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2709.823851] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2712.688039] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2717.319301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2717.573970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2721.474901] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2725.200703] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2735.027545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2742.240396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2742.256180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.266615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2745.322609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2745.328727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2745.339995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2747.078257] LustreError: 108328:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2747.241327] Lustre: server umount lustre-OST0000 complete [ 2753.510724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2753.513774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2753.530264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2753.532856] Lustre: Skipped 1 previous similar message [ 2755.556765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2755.557450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2756.703952] LustreError: 108530:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2756.714984] LustreError: 108530:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2756.846127] Lustre: server umount lustre-MDT0000 complete [ 2760.035717] LustreError: 106044:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763512785 with bad export cookie 11612371800086529025 [ 2760.040376] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2760.076791] LustreError: 106044:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2760.225382] LustreError: 108732:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2760.238568] LustreError: 108732:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2760.499614] Lustre: server umount lustre-MDT0001 complete [ 2770.298710] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2772.557806] Key type lgssc unregistered [ 2772.825964] LNet: 109265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2772.834435] LNetError: 109265:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2772.861845] LNet: Removed LNI 192.168.204.115@tcp [ 2773.630653] Key type .llcrypt unregistered [ 2773.633179] Key type ._llcrypt unregistered [ 2785.584255] Key type ._llcrypt registered [ 2785.585904] Key type .llcrypt registered [ 2785.686053] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2786.589378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2786.610846] alg: No test for adler32 (adler32-zlib) [ 2787.703672] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 2787.882150] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2789.535186] Key type lgssc registered [ 2790.647888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2795.199381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2795.203541] LustreError: 110301:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2795.224190] LustreError: 110309:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2795.230779] LustreError: 110301:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4977 [ 2795.238135] LustreError: 110301:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2795.250702] LustreError: 110301:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2795.253979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2796.707699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.730707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2802.964635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2806.355886] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2810.605331] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing lsmod [ 2813.122458] LustreError: 111477:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2813.156621] LustreError: 111485:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2813.167836] LustreError: 111477:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4966 [ 2813.170991] LustreError: 111477:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2813.186211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2813.187545] LustreError: 111477:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2813.599850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2818.531911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2818.538134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2818.576331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2819.046125] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2822.379795] LustreError: 111671:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2822.464770] Lustre: server umount lustre-MDT0000 complete [ 2824.160294] LustreError: 110979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2824.169436] LustreError: 110979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2825.394400] LustreError: 110328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763512850 with bad export cookie 9576841344468498038 [ 2825.401602] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2825.409876] LustreError: 110328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2825.535747] LustreError: 111875:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2825.540895] LustreError: 111875:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2825.705915] Lustre: server umount lustre-MDT0001 complete [ 2838.530799] LustreError: 112076:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2838.542308] LustreError: 112076:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2838.640752] Lustre: server umount lustre-OST0000 complete [ 2845.255325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2845.687522] LustreError: 112554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2845.751615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2848.806265] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2850.787098] LustreError: 112554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2854.881208] LustreError: 112553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2855.698634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2855.981438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2859.472407] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2864.962862] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2868.925502] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2875.886129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2876.212882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2880.833638] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2884.938118] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2890.542587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2900.960717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2900.965767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2900.990943] Lustre: Skipped 2 previous similar messages [ 2901.016856] Lustre: Skipped 4 previous similar messages [ 2905.461113] LustreError: 114822:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2905.464210] LustreError: 114822:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2905.608155] Lustre: server umount lustre-OST0000 complete [ 2911.201875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2911.226817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2911.233515] Lustre: Skipped 1 previous similar message [ 2915.331371] LustreError: 115024:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2915.352231] LustreError: 115024:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2915.483531] Lustre: server umount lustre-MDT0000 complete [ 2916.320227] LustreError: 112549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2916.365136] LustreError: 112549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2919.137842] LustreError: 112535:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763512944 with bad export cookie 9576841344468498717 [ 2919.157326] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2919.450786] Lustre: server umount lustre-MDT0001 complete [ 2928.567423] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2931.537533] Key type lgssc unregistered [ 2931.847949] LNet: 115758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2931.860787] LNetError: 115758:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2931.896388] LNet: Removed LNI 192.168.204.115@tcp [ 2932.799357] Key type .llcrypt unregistered [ 2932.801564] Key type ._llcrypt unregistered [ 2952.150968] Key type ._llcrypt registered [ 2952.153441] Key type .llcrypt registered [ 2952.252779] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2953.075717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2953.097816] alg: No test for adler32 (adler32-zlib) [ 2954.136721] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 2954.295833] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2955.999230] Key type lgssc registered [ 2956.997283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2963.777979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2963.796820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2965.190825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2968.228856] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2974.685524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2974.947602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2978.537610] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2982.564799] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2984.882929] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2990.423267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2990.840436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2991.878913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2995.542890] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3000.794968] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3005.867907] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3008.047804] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3010.530078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.531503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3010.535114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3010.550248] Lustre: Skipped 1 previous similar message [ 3015.652423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3015.662740] Lustre: Skipped 2 previous similar messages [ 3016.383418] LustreError: 119482:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3016.531216] Lustre: server umount lustre-OST0000 complete [ 3020.782242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3020.797539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3020.811634] Lustre: Skipped 1 previous similar message [ 3020.820890] Lustre: Skipped 1 previous similar message [ 3025.889854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3025.896893] Lustre: Skipped 1 previous similar message [ 3026.175983] LustreError: 119688:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3026.188094] LustreError: 119688:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3026.282093] Lustre: server umount lustre-MDT0000 complete [ 3029.268448] LustreError: 119689:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763513054 with bad export cookie 13168343266949653741 [ 3029.275893] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3029.286702] LustreError: 119689:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3029.474707] LustreError: 119890:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3029.483474] LustreError: 119890:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3029.650593] Lustre: server umount lustre-MDT0001 complete [ 3040.687396] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3043.153412] Key type lgssc unregistered [ 3043.388680] LNet: 120422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3043.407495] LNetError: 120422:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3043.426080] LNet: Removed LNI 192.168.204.115@tcp [ 3044.284950] Key type .llcrypt unregistered [ 3044.288968] Key type ._llcrypt unregistered [ 3058.482506] Key type ._llcrypt registered [ 3058.484301] Key type .llcrypt registered [ 3058.579172] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3059.460736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3059.511525] alg: No test for adler32 (adler32-zlib) [ 3060.514104] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 3060.687326] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3062.303291] Key type lgssc registered [ 3063.390736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3070.543669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3070.556264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3071.995428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3075.504364] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3082.213857] LustreError: 121625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.229668] LustreError: 121625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3083.201390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3083.629783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3087.223468] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3092.571503] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3095.921727] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3102.186322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3102.482354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3107.140388] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3111.766350] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3116.853144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3124.197258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3124.207384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3124.223610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3127.265698] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3127.270860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3127.272089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3127.285724] Lustre: Skipped 1 previous similar message [ 3129.299992] LustreError: 123872:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3129.507341] Lustre: server umount lustre-OST0000 complete [ 3134.945088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3134.948934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.957886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3137.504352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3137.504359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3138.806784] LustreError: 124075:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3138.814318] LustreError: 124075:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3138.899890] Lustre: server umount lustre-MDT0000 complete [ 3141.989154] LustreError: 121606:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763513167 with bad export cookie 7742671771002460375 [ 3141.993414] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3142.126960] LustreError: 124276:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3142.131682] LustreError: 124276:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3142.288167] Lustre: server umount lustre-MDT0001 complete [ 3150.389413] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3152.305594] Key type lgssc unregistered [ 3152.548811] LNet: 124809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3152.561091] LNetError: 124809:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3152.584630] LNet: Removed LNI 192.168.204.115@tcp [ 3153.277527] Key type .llcrypt unregistered [ 3153.279325] Key type ._llcrypt unregistered [ 3171.595834] Key type ._llcrypt registered [ 3171.599360] Key type .llcrypt registered [ 3171.672376] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3172.161434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3172.171722] alg: No test for adler32 (adler32-zlib) [ 3173.133702] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 3173.304685] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3174.935288] Key type lgssc registered [ 3175.833959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3181.455660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3181.475444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3182.771279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3185.480609] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3191.450460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3191.703977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3194.927808] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3199.200897] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3202.694766] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3207.656473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3207.905212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3212.101287] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3216.454303] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3227.372418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3237.492365] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3241.248841] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3241.264938] Lustre: Skipped 2 previous similar messages [ 3245.464902] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3245.469338] Lustre: Skipped 2 previous similar messages [ 3247.706325] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3247.740931] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3251.667575] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.15@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3251.672518] Lustre: Skipped 2 previous similar messages [ 3259.958348] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.15@tcp 192.168.0.[2,10]@tcp [ 3259.976631] Lustre: Skipped 1 previous similar message [ 3263.238292] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.15/24@tcp in log lustre-MDT0000 [ 3263.245428] Lustre: Skipped 2 previous similar messages [ 3269.144315] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.15/24@tcp [ 3273.697797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3273.701440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3273.712271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3273.715039] Lustre: Skipped 1 previous similar message [ 3278.816535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3278.829483] Lustre: Skipped 1 previous similar message [ 3279.169039] LustreError: 129951:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3279.256214] Lustre: server umount lustre-OST0000 complete [ 3283.937322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3283.938432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.940406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3283.950551] Lustre: Skipped 2 previous similar messages [ 3289.057072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3289.067994] Lustre: Skipped 2 previous similar messages [ 3294.176704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3294.178763] Lustre: Skipped 1 previous similar message [ 3296.223138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3296.325145] LustreError: 130152:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3296.343196] LustreError: 130152:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3296.417979] Lustre: server umount lustre-MDT0000 complete [ 3299.261814] LustreError: 126164:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763513324 with bad export cookie 15565131713595951546 [ 3299.264066] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3299.271695] LustreError: 126164:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3299.413729] LustreError: 130357:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3299.425366] LustreError: 130357:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3299.554270] Lustre: server umount lustre-MDT0001 complete [ 3306.667302] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3308.447432] Key type lgssc unregistered [ 3308.662710] LNet: 130887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3308.668348] LNetError: 130887:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3308.681699] LNet: Removed LNI 192.168.204.115@tcp [ 3309.290731] Key type .llcrypt unregistered [ 3309.293884] Key type ._llcrypt unregistered [ 3329.255926] Key type ._llcrypt registered [ 3329.257515] Key type .llcrypt registered [ 3329.350525] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3329.969217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3330.020985] alg: No test for adler32 (adler32-zlib) [ 3330.980647] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 3331.132766] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3332.759219] Key type lgssc registered [ 3333.507669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3339.872598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3339.911490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.264991] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3341.292490] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.15/24@tcp [ 3341.314363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3344.381193] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3350.708139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3351.025505] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3351.040329] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.15/24@tcp [ 3351.073957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3353.607147] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3357.122726] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3359.417893] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3364.169106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3364.518916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3367.608664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3368.578379] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3372.336333] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3385.832124] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3385.837759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3385.867166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3387.360686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3387.364533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3387.372712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.563153] LustreError: 134709:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3389.680453] Lustre: server umount lustre-OST0000 complete [ 3397.098904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3397.103980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3397.128072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3397.131046] Lustre: Skipped 1 previous similar message [ 3399.589424] LustreError: 134910:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3399.597423] LustreError: 134910:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3399.661529] Lustre: server umount lustre-MDT0000 complete [ 3402.722427] LustreError: 133784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.738225] LustreError: 133784:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3402.941364] LustreError: 132388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763513428 with bad export cookie 11727402661664312351 [ 3402.947319] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3402.949839] LustreError: 132388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3403.048471] LustreError: 135111:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3403.052736] LustreError: 135111:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3403.199841] Lustre: server umount lustre-MDT0001 complete [ 3410.223295] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3412.144693] Key type lgssc unregistered [ 3412.368633] LNet: 135644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3412.373396] LNetError: 135644:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3412.395171] LNet: Removed LNI 192.168.204.115@tcp [ 3412.958887] Key type .llcrypt unregistered [ 3412.960848] Key type ._llcrypt unregistered [ 3430.007982] Key type ._llcrypt registered [ 3430.009651] Key type .llcrypt registered [ 3430.092319] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3440.418190] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3440.932122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3440.942889] alg: No test for adler32 (adler32-zlib) [ 3441.919054] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 3442.092460] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3443.752980] Key type lgssc registered [ 3444.712948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3450.828646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3455.007734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3459.538913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3464.257963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3469.153170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3469.203786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3469.225763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3470.404870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3470.428051] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3470.488461] Lustre: lustre-MDT0000: new disk, initializing [ 3470.549073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3470.565131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3473.250548] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3480.634521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3480.683828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3480.743979] Lustre: 138994:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3480.798158] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3480.800910] Lustre: Skipped 1 previous similar message [ 3480.871502] Lustre: lustre-MDT0001: new disk, initializing [ 3480.958877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3481.000237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3481.006167] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3483.513816] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3489.597822] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3492.092021] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3496.908380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3496.964054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3497.115195] Lustre: lustre-OST0000: new disk, initializing [ 3497.118597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3497.189880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3500.663192] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3503.663919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3503.687146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3503.801286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3507.785816] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3511.277944] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3511.500809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3516.684532] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3516.877348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3519.456803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3519.461235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3519.466407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3519.479315] Lustre: Skipped 1 previous similar message [ 3519.502765] Lustre: Skipped 1 previous similar message [ 3524.580890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3524.583756] Lustre: Skipped 1 previous similar message [ 3529.700821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3533.283054] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3533.324632] LustreError: 141936:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3533.420432] Lustre: server umount lustre-OST0000 complete [ 3537.376240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3537.391229] LustreError: Skipped 1 previous similar message [ 3537.394818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3537.407702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3537.410412] Lustre: Skipped 1 previous similar message [ 3539.939063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3542.505244] LustreError: 142139:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3542.510456] LustreError: 142139:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3542.579526] Lustre: server umount lustre-MDT0000 complete [ 3545.165165] LustreError: 140741:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763513570 with bad export cookie 10996113160555125031 [ 3545.168525] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3545.170529] LustreError: 140741:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3545.267575] LustreError: 142341:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3545.272433] LustreError: 142341:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3545.394143] Lustre: server umount lustre-MDT0001 complete [ 3552.006040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3552.439176] LustreError: 142817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3552.450772] LustreError: 142817:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3552.500995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3555.416885] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3557.860872] LustreError: 142817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3561.953632] LustreError: 142817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3562.712920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3563.093032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3566.780484] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3571.112779] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3573.769209] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3579.437280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3579.768735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3584.880198] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3590.231248] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3593.177369] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3597.529693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3600.234723] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3600.393934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3607.345194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3607.406950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3607.516695] Lustre: lustre-OST0001: new disk, initializing [ 3607.526184] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3607.593233] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3611.585373] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3615.327178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3615.334376] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3615.424567] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3617.646986] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3617.801945] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3620.986600] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3621.144490] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3624.001684] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b0544570800.ost_server_uuid 50 [ 3625.074202] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b0544570800.ost_server_uuid in FULL state after 0 sec [ 3631.085615] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3631.086175] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3631.093775] LustreError: Skipped 1 previous similar message [ 3631.102301] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3631.111020] Lustre: Skipped 3 previous similar messages [ 3636.160098] LustreError: 147092:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3636.167433] LustreError: 147092:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3636.197795] LustreError: 146393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3636.213104] LustreError: 146393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3636.252215] Lustre: server umount lustre-OST0001 complete [ 3641.316047] LustreError: 145851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3641.323158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3641.338691] LustreError: 145851:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3641.352808] Lustre: Skipped 1 previous similar message [ 3641.360583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3641.368479] Lustre: Skipped 1 previous similar message [ 3645.507135] LustreError: 147296:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3645.520397] LustreError: 147296:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3645.663461] Lustre: server umount lustre-OST0000 complete [ 3650.016882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3650.029832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.038028] Lustre: Skipped 1 previous similar message [ 3661.798205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3661.805852] Lustre: Skipped 6 previous similar messages [ 3663.327174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3663.468446] LustreError: 147498:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3663.472588] LustreError: 147498:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3663.566556] Lustre: server umount lustre-MDT0000 complete [ 3666.614182] LustreError: 142841:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763513691 with bad export cookie 10996113160555125913 [ 3666.615626] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3666.636238] LustreError: 142841:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3666.949619] Lustre: server umount lustre-MDT0001 complete [ 3681.057885] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3683.241568] Key type lgssc unregistered [ 3683.498631] LNet: 148689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3683.520243] LNetError: 148689:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3684.582557] LNet: Removed LNI 192.168.204.115@tcp [ 3685.286179] Key type .llcrypt unregistered [ 3685.287905] Key type ._llcrypt unregistered [ 3714.806152] Key type ._llcrypt registered [ 3714.808183] Key type .llcrypt registered [ 3714.907688] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3725.984237] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3726.960983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3726.979173] alg: No test for adler32 (adler32-zlib) [ 3727.943696] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 3728.115993] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3729.760059] Key type lgssc registered [ 3730.627147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3737.350483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3742.065682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3746.677543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3751.789369] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3757.254099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3757.297764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3757.308452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3758.515590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3758.537649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3758.611624] Lustre: lustre-MDT0000: new disk, initializing [ 3758.692949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3758.717204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3761.815824] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3770.495800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3770.537352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3770.619143] Lustre: 152621:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3770.686280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3770.691627] Lustre: Skipped 1 previous similar message [ 3770.756333] Lustre: lustre-MDT0001: new disk, initializing [ 3770.861433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3770.906987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3770.921476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3774.008711] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3779.608582] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3782.635601] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3789.404328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3789.515507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3789.790399] Lustre: lustre-OST0000: new disk, initializing [ 3789.798759] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3789.883475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3795.176175] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3797.534353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3797.542170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3797.656075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3801.751984] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3805.311353] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3805.499258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3808.793758] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3808.970503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3812.834710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3812.841466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3812.862860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3813.353080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3813.362291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3813.369175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3813.383484] Lustre: Skipped 1 previous similar message [ 3816.440997] LustreError: 155558:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3816.588425] Lustre: server umount lustre-OST0000 complete [ 3822.050291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3822.056892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.081568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3825.874040] LustreError: 155761:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3825.878628] LustreError: 155761:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3825.941642] Lustre: server umount lustre-MDT0000 complete [ 3828.604324] LustreError: 154366:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763513853 with bad export cookie 10812091222064187132 [ 3828.606237] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3828.618030] LustreError: 154366:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3828.734569] LustreError: 155964:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3828.741705] LustreError: 155964:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3828.865476] Lustre: server umount lustre-MDT0001 complete [ 3835.188354] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 19:57:39 (1763513859) [ 3848.718822] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3855.863143] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3863.513391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3869.048971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3873.734378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3878.327833] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3883.175720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3883.244259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3883.453387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3883.481538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3883.556782] Lustre: lustre-MDT0000: new disk, initializing [ 3883.630350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3883.651084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3886.787763] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3895.444673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3895.507284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3895.614286] Lustre: 158954:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3895.661932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3895.664147] Lustre: Skipped 1 previous similar message [ 3895.722898] Lustre: lustre-MDT0001: new disk, initializing [ 3895.818251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3895.869534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3895.877838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3898.993586] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3905.071898] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3908.030624] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3913.620477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3913.670692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3913.921507] Lustre: lustre-OST0000: new disk, initializing [ 3913.927664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3914.016441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3918.045239] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3921.968731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3921.977172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3922.072945] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3923.409416] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3926.423827] Lustre: Failing over lustre-OST0000 [ 3926.466156] LustreError: 161392:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3926.469909] LustreError: 161392:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3926.551141] Lustre: server umount lustre-OST0000 complete [ 3942.816624] Lustre: 150830:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763513952/real 1763513952] req@00000000467c421a x1849178197132416/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763513968 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3942.838914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3942.845892] Lustre: Skipped 1 previous similar message [ 3943.437712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3943.742881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3943.752709] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3943.896027] Lustre: 150832:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763513952/real 1763513952] req@00000000beba8c0c x1849178197133184/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763513968 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3944.928705] Lustre: 150831:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763513954/real 1763513954] req@0000000012ce71fd x1849178197133568/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1763513970 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3944.954676] Lustre: 150831:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3945.162562] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3948.121810] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3948.774358] Lustre: 150830:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763513957/real 1763513957] req@00000000d8a467cc x1849178197134592/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763513973 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3951.732064] Lustre: Failing over lustre-MDT0000 [ 3951.743186] LustreError: 150829:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3951.849651] LustreError: 162373:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3951.852528] LustreError: 162373:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3952.025469] Lustre: server umount lustre-MDT0000 complete [ 3952.096975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3952.101045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3952.106092] Lustre: Skipped 1 previous similar message [ 3953.119309] Lustre: 150831:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763513962/real 1763513962] req@00000000d3986e69 x1849178197135488/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763513978 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3953.169337] Lustre: lustre-OST0000: Recovery over after 0:08, of 3 clients 3 recovered and 0 were evicted. [ 3953.169471] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3953.193927] Lustre: 150831:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3958.240380] LustreError: 158964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3958.259153] LustreError: 158964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3960.494532] LustreError: 160707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3963.364141] LustreError: 158959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3963.381965] LustreError: 158959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3965.617277] LustreError: 158959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3969.506171] Lustre: 150830:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763513978/real 1763513978] req@00000000d89462cc x1849178197143808/t0(0) o400->MGC192.168.204.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763513994 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3969.527245] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3970.117517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3970.727093] LustreError: 158961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3970.754549] LustreError: 158961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3979.744725] LustreError: 150829:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000dc5db9a6 x1849178197149568/t0(0) o250->MGC192.168.204.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3979.780412] LustreError: 158965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3979.791702] LustreError: 158965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3979.950851] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3980.037187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3980.081339] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3980.979085] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3983.440107] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3985.389512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3985.411054] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3985.456582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3990.500242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3990.500868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3990.505654] Lustre: Skipped 3 previous similar messages [ 3990.515023] Lustre: Skipped 3 previous similar messages [ 3991.602884] LustreError: 163263:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3991.626692] LustreError: 163263:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3991.785594] Lustre: server umount lustre-OST0000 complete [ 3995.616879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3995.626473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4001.250687] Lustre: server umount lustre-MDT0000 complete [ 4004.354483] LustreError: 158945:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763514029 with bad export cookie 10812091222064189246 [ 4004.359157] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4004.370250] LustreError: 158945:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4004.593538] Lustre: server umount lustre-MDT0001 complete [ 4013.675772] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4015.824654] Key type lgssc unregistered [ 4016.095563] LNet: 164199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4016.104696] LNetError: 164199:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4016.119561] LNet: Removed LNI 192.168.204.115@tcp [ 4016.747331] Key type .llcrypt unregistered [ 4016.749735] Key type ._llcrypt unregistered [ 4047.897319] Key type ._llcrypt registered [ 4047.899122] Key type .llcrypt registered [ 4047.995710] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4048.626110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4048.760986] alg: No test for adler32 (adler32-zlib) [ 4049.758342] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 4049.970629] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4051.609618] Key type lgssc registered [ 4052.546171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4059.061224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4059.083391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4060.449664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4063.208437] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4069.330103] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4069.857871] LustreError: 165710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4069.881062] LustreError: 165710:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4072.942662] LustreError: 165739:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4072.951149] LustreError: 165739:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4072.959374] LustreError: 165739:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 0, failed: rc = -5 [ 4073.105803] LustreError: 166263:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4073.334241] Lustre: server umount lustre-MDT0000 complete [ 4086.495240] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4094.707499] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4103.288580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4109.429468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4116.160035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4122.341232] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4128.967958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4129.028940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4129.301908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4129.345882] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4129.421846] Lustre: lustre-MDT0000: new disk, initializing [ 4129.540617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4129.560219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4133.443193] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4142.976118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4143.056809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4143.121731] Lustre: 168906:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4143.190787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4143.196847] Lustre: Skipped 1 previous similar message [ 4143.289877] Lustre: lustre-MDT0001: new disk, initializing [ 4143.409790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4143.459415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4143.495129] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4147.221568] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4153.417191] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4156.811766] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4162.601332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4162.673254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4162.876910] Lustre: lustre-OST0000: new disk, initializing [ 4162.879984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4162.952221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4168.588355] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4171.346165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4171.354102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4171.464846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4175.429119] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4179.846858] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4181.319725] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:03:25 (1763514205) [ 4404.706748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4404.712470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4404.719709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4407.281350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4407.284764] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4407.292260] Lustre: Skipped 1 previous similar message [ 4410.765447] LustreError: 172428:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4410.768642] LustreError: 172428:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4410.845625] Lustre: server umount lustre-MDT0000 complete [ 4412.384249] LustreError: 168918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4412.396343] LustreError: 168918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4413.504474] LustreError: 168898:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763514438 with bad export cookie 6370257993633305810 [ 4413.507099] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4413.511513] LustreError: 168898:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4417.505154] LustreError: 168914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4417.508938] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4417.544635] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4417.556297] Lustre: Skipped 2 previous similar messages [ 4419.675317] LustreError: 172629:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4419.677982] LustreError: 172629:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4419.740983] Lustre: server umount lustre-MDT0001 complete [ 4428.337318] LustreError: 172832:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4428.352106] LustreError: 172832:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4428.516428] Lustre: server umount lustre-OST0000 complete [ 4436.212492] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4442.841097] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4451.141971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4456.758903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4461.496172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4466.155854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4471.428802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4471.473796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4471.676871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4471.718717] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4471.833419] Lustre: lustre-MDT0000: new disk, initializing [ 4471.899301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4471.909912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4474.990686] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4483.903368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4483.983579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4484.055138] Lustre: 175175:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4484.106950] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4484.109889] Lustre: Skipped 1 previous similar message [ 4484.166175] Lustre: lustre-MDT0001: new disk, initializing [ 4484.262631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4484.290174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4484.303935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4487.255343] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4493.594196] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4496.532746] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4502.581613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4502.661926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4502.980953] Lustre: lustre-OST0000: new disk, initializing [ 4502.991716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4503.107212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4507.961661] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4511.772267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4511.779117] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4511.870338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4513.917906] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4517.156105] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4517.384778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4521.080512] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4521.295770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4527.074196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4527.077751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4527.109040] Lustre: Skipped 1 previous similar message [ 4527.124485] Lustre: Skipped 2 previous similar messages [ 4529.218552] LustreError: 178111:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4529.231796] LustreError: 178111:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4529.452295] Lustre: server umount lustre-OST0000 complete [ 4535.776428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4535.779974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4535.812665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4539.126810] LustreError: 178313:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4539.131544] LustreError: 178313:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4539.190900] Lustre: server umount lustre-MDT0000 complete [ 4542.333249] LustreError: 176919:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763514567 with bad export cookie 6370257993633464640 [ 4542.339076] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4542.360399] LustreError: 176919:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4542.623113] Lustre: server umount lustre-MDT0001 complete [ 4548.980333] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:09:33 (1763514573) [ 4560.938387] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4567.204209] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4574.444226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4579.171686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4584.703333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4590.966606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4598.054245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4598.123378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4598.395655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4598.439605] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4598.509636] Lustre: lustre-MDT0000: new disk, initializing [ 4598.593045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4598.637599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4602.312489] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4612.219897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4612.281404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4612.346841] Lustre: 181506:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4612.605524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4617.230355] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4624.061836] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4628.020434] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4634.678174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4634.745545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4635.091576] Lustre: lustre-OST0000: new disk, initializing [ 4635.095545] Lustre: Skipped 1 previous similar message [ 4635.106691] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4635.109626] Lustre: Skipped 2 previous similar messages [ 4640.424894] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4643.901131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4643.905689] Lustre: Skipped 1 previous similar message [ 4643.911171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4644.018812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4647.406895] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4654.570719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4654.573794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4654.580224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4654.588441] Lustre: Skipped 2 previous similar messages [ 4654.594647] Lustre: Skipped 2 previous similar messages [ 4659.198159] LustreError: 184053:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4659.212556] LustreError: 184053:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4659.336900] Lustre: server umount lustre-OST0000 complete [ 4663.777240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4663.794490] LustreError: Skipped 1 previous similar message [ 4668.652839] Lustre: server umount lustre-MDT0000 complete [ 4669.921231] LustreError: 181518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4669.945973] LustreError: 181518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4672.403488] LustreError: 181498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763514697 with bad export cookie 6370257993633465515 [ 4672.408241] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4672.425110] LustreError: 181498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4672.792812] Lustre: server umount lustre-MDT0001 complete [ 4679.286222] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:11:44 (1763514704) [ 4693.151972] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4701.017394] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4710.257133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4716.734302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4722.521720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4728.579860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4735.862563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4735.951215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4736.269898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4736.303619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4736.363668] Lustre: lustre-MDT0000: new disk, initializing [ 4736.434149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4736.437180] Lustre: Skipped 2 previous similar messages [ 4736.464780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4739.940552] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4749.630531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4749.695099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4749.803485] Lustre: 187449:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4750.107910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4754.143405] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4760.830558] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4764.515442] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4771.055423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4771.155568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4776.673762] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4779.177971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4783.404724] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4789.733833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4789.734350] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4789.738248] LustreError: Skipped 1 previous similar message [ 4789.740741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4789.746706] Lustre: Skipped 3 previous similar messages [ 4789.752492] Lustre: Skipped 4 previous similar messages [ 4791.231885] LustreError: 189997:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4791.245347] LustreError: 189997:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4791.346106] Lustre: server umount lustre-OST0000 complete [ 4800.772403] Lustre: server umount lustre-MDT0000 complete [ 4803.943837] LustreError: 189204:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763514829 with bad export cookie 6370257993633466607 [ 4803.947147] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4803.950613] LustreError: 189204:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4804.193931] Lustre: server umount lustre-MDT0001 complete [ 4812.473785] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4814.982247] Key type lgssc unregistered [ 4815.255764] LNet: 190936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4815.273139] LNetError: 190936:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4816.296431] LNet: Removed LNI 192.168.204.115@tcp [ 4817.278659] Key type .llcrypt unregistered [ 4817.280372] Key type ._llcrypt unregistered [ 4838.348193] Key type ._llcrypt registered [ 4838.349737] Key type .llcrypt registered [ 4838.441280] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4839.243911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4839.375379] alg: No test for adler32 (adler32-zlib) [ 4840.344792] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 4840.581914] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4842.271271] Key type lgssc registered [ 4842.990284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4848.906192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4848.927327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4850.329859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4853.079958] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4859.372583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4859.694886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4862.735292] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4867.278358] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4870.364737] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4876.350714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4876.644278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4881.957645] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4886.513365] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4895.102710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4914.658241] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4914.664751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4914.673297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4915.682594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4915.686266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4915.711546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4916.920909] LustreError: 194593:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4917.071446] Lustre: server umount lustre-OST0000 complete [ 4920.805502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4920.815242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4920.817577] Lustre: Skipped 1 previous similar message [ 4925.704894] LustreError: 194794:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4925.707888] LustreError: 194794:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4925.753818] Lustre: server umount lustre-MDT0000 complete [ 4925.920535] LustreError: 192321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4925.927467] LustreError: 192321:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4927.812227] LustreError: 192306:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763514953 with bad export cookie 13490676343433435651 [ 4927.812894] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4927.821840] LustreError: 192306:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4927.926662] LustreError: 194996:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4927.929591] LustreError: 194996:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4928.055340] Lustre: server umount lustre-MDT0001 complete [ 4934.004242] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4936.097615] Key type lgssc unregistered [ 4936.343652] LNet: 195529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4936.350165] LNetError: 195529:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4936.360871] LNet: Removed LNI 192.168.204.115@tcp [ 4936.834853] Key type .llcrypt unregistered [ 4936.836730] Key type ._llcrypt unregistered [ 4953.284205] Key type ._llcrypt registered [ 4953.285876] Key type .llcrypt registered [ 4953.375506] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4954.020748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4954.032438] alg: No test for adler32 (adler32-zlib) [ 4955.068789] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 4955.249321] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4956.895721] Key type lgssc registered [ 4957.825867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4964.480050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4964.498425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4965.997460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4969.528709] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4976.620233] LustreError: 196905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4976.646563] LustreError: 196905:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4977.273811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4977.943715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4981.729309] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4986.569969] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4989.307442] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4995.166731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4995.421368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4999.541593] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5003.248900] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5011.822921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5019.105819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5019.109251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5019.135060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5020.354059] Lustre: lustre-OST0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 5022.176882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5022.178325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5022.186294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5024.954594] LustreError: 199173:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5025.067181] Lustre: server umount lustre-OST0000 complete [ 5029.036639] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8b05527f7000.ost_server_uuid 50 [ 5030.205850] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5042.390668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5042.411093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5042.413745] Lustre: Skipped 2 previous similar messages [ 5047.521099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5048.559991] LustreError: 199540:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5048.567302] LustreError: 199540:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5048.634370] Lustre: server umount lustre-MDT0000 complete [ 5051.598970] LustreError: 196886:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763515076 with bad export cookie 3362104981528373660 [ 5051.603791] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5051.619270] LustreError: 196886:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5051.715299] LustreError: 199741:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5051.717498] LustreError: 199741:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5051.833094] Lustre: server umount lustre-MDT0001 complete [ 5058.253227] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:18:02 (1763515082) [ 5063.275306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5063.537267] LustreError: 200414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5063.591350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5066.641693] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5068.773127] LustreError: 200414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5072.865757] LustreError: 200414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5073.276684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5073.546291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5076.197360] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5079.799945] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5082.644374] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5087.884237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5088.102550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5089.154519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5091.865315] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5095.717267] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5100.762324] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5100.823889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5100.959548] Lustre: lustre-OST0001: new disk, initializing [ 5100.968678] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5104.826139] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5108.290533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5108.295223] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5108.384776] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5109.995276] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5113.831465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5113.835164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5113.838723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5113.843689] Lustre: Skipped 2 previous similar messages [ 5113.846226] Lustre: Skipped 1 previous similar message [ 5126.112162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5126.159456] LustreError: 203618:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5126.167441] LustreError: 203618:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5126.218717] Lustre: server umount lustre-OST0000 complete [ 5129.189102] LustreError: 202011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5129.197447] LustreError: 202011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5129.736304] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5129.897144] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5132.463269] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5132.632928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5139.427153] LustreError: 202013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5139.442243] LustreError: 202013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5144.552955] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5144.554884] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5144.562352] LustreError: Skipped 1 previous similar message [ 5144.565836] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5144.570792] Lustre: Skipped 1 previous similar message [ 5144.581506] Lustre: Skipped 8 previous similar messages [ 5149.890295] LustreError: 204322:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5149.899380] LustreError: 204322:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5150.014868] Lustre: server umount lustre-OST0001 complete [ 5154.785602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5158.814612] LustreError: 204523:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5158.819928] LustreError: 204523:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5158.911161] Lustre: server umount lustre-MDT0000 complete [ 5159.905883] LustreError: 200415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5159.917745] LustreError: 200415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5161.632151] LustreError: 200394:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763515186 with bad export cookie 3362104981528374731 [ 5161.637132] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5161.639381] LustreError: 200394:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5161.855705] Lustre: server umount lustre-MDT0001 complete [ 5165.553595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5168.132945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5170.841520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5173.764414] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5178.855114] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 20:20:03 (1763515203) [ 5184.328995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5184.675580] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5184.705647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5184.866902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5184.869718] Lustre: Skipped 1 previous similar message [ 5187.915652] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5194.178466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5194.271068] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5194.298736] Lustre: 206216:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5197.307443] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5200.822933] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5203.653804] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5208.957260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5209.116386] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5212.802916] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5215.319657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5216.002985] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5220.579949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5220.649841] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5220.825218] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5220.829821] Lustre: Skipped 2 previous similar messages [ 5224.668235] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5228.052881] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5230.561150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5230.562967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5230.571480] LustreError: Skipped 1 previous similar message [ 5230.572423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5230.572428] Lustre: Skipped 5 previous similar messages [ 5230.587802] Lustre: Skipped 2 previous similar messages [ 5236.030136] LustreError: 209413:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5236.070928] LustreError: 209413:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5236.100433] Lustre: server umount lustre-OST0000 complete [ 5238.754089] LustreError: 207846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5238.762675] LustreError: 207846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5248.992272] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5254.853779] Lustre: server umount lustre-OST0001 complete [ 5263.613560] Lustre: server umount lustre-MDT0000 complete [ 5265.747918] LustreError: 206208:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763515290 with bad export cookie 3362104981528376138 [ 5265.751425] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5265.754752] LustreError: 206208:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5265.939983] Lustre: server umount lustre-MDT0001 complete [ 5269.300435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5271.728969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5273.996031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5276.518922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5280.600251] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 20:21:45 (1763515305) [ 5291.295543] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 5299.078522] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5306.071759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5310.922799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5315.781158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5320.897907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5326.392850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5326.456210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5326.658391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5326.695988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5326.754403] Lustre: lustre-MDT0000: new disk, initializing [ 5326.820136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5326.834619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5329.763592] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5338.036402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5338.119121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5338.183787] Lustre: 213852:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5338.237245] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5338.240269] Lustre: Skipped 1 previous similar message [ 5338.290854] Lustre: lustre-MDT0001: new disk, initializing [ 5338.408964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5338.413593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5341.551950] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5347.232794] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5350.065710] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5354.770258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5354.824830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5354.999726] Lustre: lustre-OST0000: new disk, initializing [ 5355.002844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5358.929242] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5363.956827] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5364.772613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5364.777422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5364.838364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5366.698324] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5366.867964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5369.802251] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5369.964495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5374.945649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5374.945994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5374.954557] LustreError: Skipped 2 previous similar messages [ 5374.959222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5374.960404] Lustre: Skipped 5 previous similar messages [ 5374.970851] Lustre: Skipped 11 previous similar messages [ 5377.460340] LustreError: 216791:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5377.462354] LustreError: 216791:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5377.589821] Lustre: server umount lustre-OST0000 complete [ 5386.506155] Lustre: server umount lustre-MDT0000 complete [ 5389.194768] LustreError: 215598:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763515414 with bad export cookie 3362104981528377692 [ 5389.198501] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5389.204291] LustreError: 215598:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5389.458591] Lustre: server umount lustre-MDT0001 complete [ 5396.113139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5396.433364] LustreError: 217671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5396.451774] LustreError: 217671:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5399.412055] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5405.471767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5408.710879] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5412.583532] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5414.934681] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5419.809517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5424.353440] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5427.499374] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5429.696536] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5429.850105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5432.120273] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5432.279858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5435.360951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5435.366876] LustreError: Skipped 1 previous similar message [ 5439.497102] Lustre: server umount lustre-OST0000 complete [ 5443.523300] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5446.174880] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5472.151631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5472.413946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5472.417469] Lustre: Skipped 5 previous similar messages [ 5475.515276] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.527111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.536075] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5475.850456] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5479.810688] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5493.945223] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec [ 5498.707774] LustreError: 222075:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5498.710822] LustreError: 222075:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 5498.832965] Lustre: server umount lustre-OST0000 complete [ 5502.944310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5502.948136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5502.954584] Lustre: Skipped 4 previous similar messages [ 5502.960757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5502.963761] Lustre: Skipped 5 previous similar messages [ 5511.726628] Lustre: server umount lustre-MDT0000 complete [ 5514.240918] LustreError: 217652:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763515539 with bad export cookie 3362104981528378581 [ 5514.245167] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5514.258754] LustreError: 217652:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5524.602149] Lustre: server umount lustre-MDT0001 complete [ 5530.167435] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 20:25:55 (1763515555) [ 5534.955959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5535.255720] LustreError: 223152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5535.269235] LustreError: 223152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5537.939718] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5543.969653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5546.955527] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5550.396689] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5552.704214] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5557.174085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5561.065525] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5564.347790] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5566.691920] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5566.843831] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5569.415824] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5570.626770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 5575.549965] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5575.597377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5575.697208] Lustre: lustre-OST0001: new disk, initializing [ 5575.700159] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5578.620813] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5582.917029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5582.927420] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5582.975489] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5583.229556] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5585.596662] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5585.729719] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5588.331399] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5588.460492] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5595.756990] Lustre: server umount lustre-OST0001 complete [ 5600.126940] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5600.277136] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5602.854962] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5602.974571] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5628.775743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5632.557538] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5634.022501] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5634.035285] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5634.037263] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5634.055654] Lustre: Skipped 2 previous similar messages [ 5635.940861] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5636.960164] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5639.137213] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5639.142476] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5639.147586] LustreError: Skipped 1 previous similar message [ 5639.153370] Lustre: Skipped 3 previous similar messages [ 5644.343352] LustreError: 229016:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5644.350405] LustreError: 229016:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 5644.388683] Lustre: server umount lustre-OST0001 complete [ 5663.920840] Lustre: server umount lustre-OST0000 complete [ 5680.095197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5680.193683] Lustre: server umount lustre-MDT0000 complete [ 5682.180795] LustreError: 223133:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763515707 with bad export cookie 3362104981528379547 [ 5682.184099] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5682.187801] LustreError: 223133:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5688.489933] Lustre: server umount lustre-MDT0001 complete [ 5691.952311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5694.314407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5696.618750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5698.977289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5703.115694] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:28:48 (1763515728) [ 5706.759924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5706.933775] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5706.955521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5709.120533] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5714.215160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5714.324134] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5714.345383] Lustre: 231137:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5716.923620] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5720.032651] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5722.068773] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5725.825759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5725.954822] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5729.476426] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5732.656623] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5738.073981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5738.133033] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5738.276340] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5738.284064] Lustre: Skipped 8 previous similar messages [ 5741.088251] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5743.721249] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5747.801616] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 5749.772157] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5749.887549] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5751.702578] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5751.802730] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5753.660156] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b05487b9000.ost_server_uuid 50 [ 5754.563061] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b05487b9000.ost_server_uuid in FULL state after 0 sec [ 5755.152858] Lustre: Permanently deactivating lustre-OST0001 [ 5755.163834] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5756.722968] Lustre: Permanently reactivating lustre-OST0001 [ 5759.466848] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5759.470227] Lustre: Skipped 22 previous similar messages [ 5764.894683] Lustre: server umount lustre-OST0001 complete [ 5773.401153] Lustre: server umount lustre-OST0000 complete [ 5781.832252] Lustre: server umount lustre-MDT0000 complete [ 5784.049606] LustreError: 232772:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763515809 with bad export cookie 3362104981528380835 [ 5784.050633] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5784.055613] LustreError: 232772:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5784.275243] Lustre: server umount lustre-MDT0001 complete [ 5787.299917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5789.322391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5791.325240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5793.236233] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5796.695257] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:30:21 (1763515821) [ 5797.226309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5800.740873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5800.917204] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5800.941542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5800.946746] Lustre: Skipped 5 previous similar messages [ 5803.523804] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5808.677774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5808.754458] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5808.780060] Lustre: 237277:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5811.208950] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5814.057466] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5816.063158] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5819.844517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5819.888815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5820.012274] Lustre: Permanently deactivating lustre-OST0000 [ 5820.020060] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5823.344165] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5826.527729] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5827.617806] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5830.912212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5833.111139] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5834.533715] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5837.552300] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5838.800756] Lustre: Permanently reactivating lustre-OST0000 [ 5838.816268] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5838.820546] Lustre: Skipped 2 previous similar messages [ 5839.459461] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5839.496924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5857.440552] Lustre: server umount lustre-OST0001 complete [ 5858.793467] LustreError: 239524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5858.799551] LustreError: 239524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 5865.610412] Lustre: server umount lustre-OST0000 complete [ 5873.879772] Lustre: server umount lustre-MDT0000 complete [ 5876.003890] LustreError: 240425:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763515901 with bad export cookie 3362104981528382480 [ 5876.008109] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5876.016037] LustreError: 240425:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5876.185951] Lustre: server umount lustre-MDT0001 complete [ 5882.033839] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5883.524364] Key type lgssc unregistered [ 5883.700461] LNet: 241690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5883.705988] LNetError: 241690:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5883.715434] LNet: Removed LNI 192.168.204.115@tcp [ 5884.190821] Key type .llcrypt unregistered [ 5884.192223] Key type ._llcrypt unregistered [ 5896.365064] Key type ._llcrypt registered [ 5896.367446] Key type .llcrypt registered [ 5896.418718] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5897.022650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5897.046300] alg: No test for adler32 (adler32-zlib) [ 5897.956143] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 5898.124536] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5899.743217] Key type lgssc registered [ 5900.469957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5902.480713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5906.504346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5906.514075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5907.783423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5909.722859] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5914.069031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5914.115036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5914.189087] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5914.207044] Lustre: 243106:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5914.213884] Lustre: Permanently deactivating lustre-MDT0001 [ 5914.223713] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5914.378386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5916.164277] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5916.168792] LustreError: 243148:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5916.177225] LustreError: 243148:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5916.183524] LustreError: 243148:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 5916.196733] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5918.800197] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5920.780508] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5924.450330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5924.643276] LustreError: 244736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5924.653548] LustreError: 244736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5924.729682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5928.201082] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5929.958747] LustreError: 244736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5929.965131] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5931.230647] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5932.004052] LustreError: 244734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5932.013397] LustreError: 244734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5932.015599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5935.255599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5935.373692] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5938.571901] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5940.725934] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5940.726729] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5941.240340] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5943.106736] Lustre: Permanently reactivating lustre-MDT0001 [ 5943.116651] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5943.119788] Lustre: Skipped 2 previous similar messages [ 5951.680706] Lustre: lustre-MDT0001: Client a1ee5774-f203-41c3-8447-43176a93a72c (at 192.168.204.15@tcp) reconnecting [ 5954.332700] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5955.130902] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5957.808317] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5957.917258] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5959.988370] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5960.126946] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5961.342854] Lustre: Permanently deactivating lustre-MDT0001 [ 5961.351599] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5961.354237] Lustre: Skipped 2 previous similar messages [ 5967.885441] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5970.913682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5970.917543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5970.924131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5971.429116] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5971.429197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5971.435300] Lustre: Skipped 2 previous similar messages [ 5971.439594] Lustre: Skipped 2 previous similar messages [ 5975.507063] LustreError: 247183:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5975.569141] Lustre: server umount lustre-MDT0000 complete [ 5976.546964] LustreError: 243117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5976.555942] LustreError: 243117:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5977.437597] LustreError: 243098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516002 with bad export cookie 9830294516445423950 [ 5977.441300] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5977.443254] LustreError: 243098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5977.535671] LustreError: 247386:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5977.538521] LustreError: 247386:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5977.618435] Lustre: server umount lustre-MDT0001 complete [ 5985.599128] LustreError: 247586:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5985.605101] LustreError: 247586:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5985.642467] Lustre: server umount lustre-OST0000 complete [ 5990.959840] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:33:36 (1763516016) [ 5995.420530] LustreError: 248590:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5995.423636] LustreError: 248590:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5995.496485] Lustre: server umount lustre-OST0001 complete [ 5998.067490] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6002.033259] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6006.483098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6009.373386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6012.437701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6015.241357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6018.189877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6018.220268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6018.324291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6018.328685] Lustre: Skipped 2 previous similar messages [ 6018.342294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6018.384081] Lustre: lustre-MDT0000: new disk, initializing [ 6018.421935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6018.431252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6020.055848] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6025.011876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6025.043442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6025.087683] Lustre: 250773:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6025.116406] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6025.119142] Lustre: Skipped 1 previous similar message [ 6025.160400] Lustre: lustre-MDT0001: new disk, initializing [ 6025.218336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6025.222891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6026.833320] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6030.403434] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6031.867802] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6034.670522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6034.703212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6034.830704] Lustre: lustre-OST0000: new disk, initializing [ 6034.834819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6034.873646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6034.876590] Lustre: Skipped 1 previous similar message [ 6035.716171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6035.720834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6035.773639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6037.387852] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6040.941053] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6041.708566] LustreError: 250780:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6046.485391] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6046.526781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6046.601443] Lustre: lustre-OST0001: new disk, initializing [ 6046.604897] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6049.122165] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6051.799072] LustreError: 250780:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6051.813447] LustreError: 250809:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6051.850607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6051.855059] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6051.891551] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6052.687497] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6053.307389] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6061.903229] LustreError: 250809:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6063.288618] Lustre: lustre-OST0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 6063.291672] Lustre: Skipped 1 previous similar message [ 6067.171240] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6067.173916] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6067.179928] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6067.182119] Lustre: Skipped 1 previous similar message [ 6068.717084] LustreError: 254272:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6068.719605] LustreError: 254272:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6068.741826] Lustre: server umount lustre-OST0001 complete [ 6072.292245] LustreError: 252502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6072.299034] LustreError: 252502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6082.017295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6082.017719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6082.020812] LustreError: Skipped 2 previous similar messages [ 6082.025950] Lustre: Skipped 1 previous similar message [ 6082.028579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6082.031986] Lustre: Skipped 2 previous similar messages [ 6082.530370] LustreError: 252505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6082.537558] LustreError: 252505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6087.286113] LustreError: 254495:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6087.289940] LustreError: 254495:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6087.347485] Lustre: server umount lustre-OST0000 complete [ 6091.744557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6091.747412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6091.752318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6095.137296] Lustre: server umount lustre-MDT0000 complete [ 6096.886527] LustreError: 250766:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516122 with bad export cookie 9830294516445426757 [ 6096.889348] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6096.891629] LustreError: 250766:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6111.200125] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6111.236737] LustreError: 254897:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6111.239459] LustreError: 254897:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6111.311854] Lustre: server umount lustre-MDT0001 complete [ 6115.553506] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6116.732506] Key type lgssc unregistered [ 6116.874465] LNet: 255430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6116.878302] LNetError: 255430:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6116.889889] LNet: Removed LNI 192.168.204.115@tcp [ 6117.215234] Key type .llcrypt unregistered [ 6117.216511] Key type ._llcrypt unregistered [ 6130.789565] Key type ._llcrypt registered [ 6130.791395] Key type .llcrypt registered [ 6130.836357] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6137.803900] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6138.276955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6138.290472] alg: No test for adler32 (adler32-zlib) [ 6139.187871] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 6139.285849] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6140.879890] Key type lgssc registered [ 6141.317740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6144.471457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6146.914570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6149.414652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6151.992664] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6154.625172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6154.651206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6154.658549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6155.766771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6155.789582] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6155.836931] Lustre: lustre-MDT0000: new disk, initializing [ 6155.885039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6155.897207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6157.499493] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6162.155408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6162.178440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6162.215736] Lustre: 259352:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6162.243280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6162.245866] Lustre: Skipped 1 previous similar message [ 6162.284417] Lustre: lustre-MDT0001: new disk, initializing [ 6162.323047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6162.338269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6162.342558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6163.748502] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6167.122298] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6168.511150] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6171.093083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6171.128953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6171.249778] Lustre: lustre-OST0000: new disk, initializing [ 6171.252603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6171.294876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6173.667856] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6176.766950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6176.772202] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6176.810485] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6177.034205] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6178.501715] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6178.582638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6180.062991] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6180.157289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6181.859122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6181.859175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6181.862347] LustreError: Skipped 1 previous similar message [ 6181.868604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6181.871371] Lustre: Skipped 1 previous similar message [ 6182.961847] LustreError: 262291:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6183.015904] Lustre: server umount lustre-OST0000 complete [ 6186.977512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6186.978227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6186.983062] Lustre: Skipped 2 previous similar messages [ 6186.987862] Lustre: Skipped 1 previous similar message [ 6190.596407] LustreError: 262491:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6190.599857] LustreError: 262491:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6190.642321] Lustre: server umount lustre-MDT0000 complete [ 6192.187773] LustreError: 260134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516217 with bad export cookie 13953783368520542855 [ 6192.191338] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6192.193556] LustreError: 260134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6192.261973] LustreError: 262693:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6192.264094] LustreError: 262693:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6192.343018] Lustre: server umount lustre-MDT0001 complete [ 6195.895498] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 20:37:01 (1763516221) [ 6198.567412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6198.763592] LustreError: 263362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6198.771826] LustreError: 263362:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6198.793448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6200.330813] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6203.872352] LustreError: 263362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6204.074513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6204.279880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6205.995723] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6208.396337] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6210.029482] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6212.814420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6213.017483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6215.421969] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6218.015294] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6264.290831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6264.291455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6264.296972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6264.303138] Lustre: Skipped 1 previous similar message [ 6268.403112] LustreError: 265684:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6268.406076] LustreError: 265684:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6268.466366] Lustre: server umount lustre-OST0000 complete [ 6270.370815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6275.720763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6275.888562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6278.243860] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6280.550868] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6284.068779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.074800] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.074891] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6284.082409] Lustre: Skipped 1 previous similar message [ 6314.977017] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6314.977676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6314.978488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6314.987099] Lustre: Skipped 2 previous similar messages [ 6320.097033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6320.099707] Lustre: Skipped 2 previous similar messages [ 6320.623095] LustreError: 267347:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6320.625316] LustreError: 267347:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6320.680422] Lustre: server umount lustre-OST0000 complete [ 6325.217146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6328.355092] Lustre: server umount lustre-MDT0000 complete [ 6329.758802] LustreError: 263341:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516354 with bad export cookie 13953783368520543723 [ 6329.762941] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6329.764237] LustreError: 263341:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6329.847560] LustreError: 267750:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6329.850483] LustreError: 267750:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6329.929272] Lustre: server umount lustre-MDT0001 complete [ 6333.872181] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6335.023687] Key type lgssc unregistered [ 6335.174209] LNet: 268280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6335.178053] LNetError: 268280:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6335.185826] LNet: Removed LNI 192.168.204.115@tcp [ 6335.522341] Key type .llcrypt unregistered [ 6335.524093] Key type ._llcrypt unregistered [ 6345.250667] Key type ._llcrypt registered [ 6345.252007] Key type .llcrypt registered [ 6345.292385] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6345.706855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6345.714462] alg: No test for adler32 (adler32-zlib) [ 6346.578902] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 6346.672995] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6348.263227] Key type lgssc registered [ 6348.670639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6351.635704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6351.645284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6352.839134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6354.280756] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6357.421749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6357.550498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6358.976586] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6360.926248] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6362.181623] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6364.631167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6364.760205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6366.847330] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6368.715339] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6378.086413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6388.705436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6388.707564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6388.713664] Lustre: Skipped 1 previous similar message [ 6388.718909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6393.828159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6393.830993] Lustre: Skipped 2 previous similar messages [ 6394.222306] LustreError: 272691:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6394.277826] Lustre: server umount lustre-OST0000 complete [ 6398.432170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6398.435703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6398.442188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6404.064919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6404.067783] Lustre: Skipped 2 previous similar messages [ 6409.184340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6409.187156] Lustre: Skipped 1 previous similar message [ 6410.207211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6410.248215] LustreError: 272892:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6410.251328] LustreError: 272892:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6410.283216] Lustre: server umount lustre-MDT0000 complete [ 6411.767372] LustreError: 269634:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516436 with bad export cookie 6562181698960030270 [ 6411.768267] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6411.771918] LustreError: 269634:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6411.842119] LustreError: 273095:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6411.844987] LustreError: 273095:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6411.902787] Lustre: server umount lustre-MDT0001 complete [ 6416.072528] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6417.300228] Key type lgssc unregistered [ 6417.443411] LNet: 273627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6417.446723] LNetError: 273627:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6417.455389] LNet: Removed LNI 192.168.204.115@tcp [ 6417.810263] Key type .llcrypt unregistered [ 6417.811917] Key type ._llcrypt unregistered [ 6425.720342] Key type ._llcrypt registered [ 6425.721847] Key type .llcrypt registered [ 6425.765337] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6426.203543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6426.265069] alg: No test for adler32 (adler32-zlib) [ 6427.128830] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 6427.220487] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6428.807219] Key type lgssc registered [ 6429.247604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6432.416837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6432.428036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6433.655365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6435.263726] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6438.808779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6438.969179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6440.525430] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6442.644504] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6444.116566] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6446.930087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6447.104873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6449.470368] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6451.711489] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6458.409875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6468.576720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6468.578725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6468.586565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6468.588479] Lustre: Skipped 1 previous similar message [ 6472.047139] LustreError: 277217:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6472.099550] Lustre: server umount lustre-OST0000 complete [ 6473.696448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6473.697252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6473.701783] Lustre: Skipped 2 previous similar messages [ 6478.816585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6478.820881] Lustre: Skipped 2 previous similar messages [ 6479.813153] LustreError: 277419:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6479.815240] LustreError: 277419:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6479.852806] Lustre: server umount lustre-MDT0000 complete [ 6481.408666] LustreError: 274786:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516506 with bad export cookie 11886197248851115557 [ 6481.411964] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6481.414873] LustreError: 274786:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6481.504791] LustreError: 277622:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6481.507877] LustreError: 277622:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6481.581036] Lustre: server umount lustre-MDT0001 complete [ 6486.198106] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6487.555128] Key type lgssc unregistered [ 6487.726609] LNet: 278154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6487.733110] LNetError: 278154:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6487.745949] LNet: Removed LNI 192.168.204.115@tcp [ 6488.115376] Key type .llcrypt unregistered [ 6488.117194] Key type ._llcrypt unregistered [ 6499.449824] Key type ._llcrypt registered [ 6499.450898] Key type .llcrypt registered [ 6499.490290] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6499.835482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6499.897606] alg: No test for adler32 (adler32-zlib) [ 6500.775911] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 6500.876449] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6502.463253] Key type lgssc registered [ 6502.919091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6505.976094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6505.983757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6507.176202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6508.714978] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6511.997628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6512.143108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6513.686392] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6515.803032] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6517.227704] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6519.843710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6519.972597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6522.065832] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6524.258995] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6533.290495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6547.937160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6547.940460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6547.946303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6548.961517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6548.962821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6548.963131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6550.767078] LustreError: 282567:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6550.805368] Lustre: server umount lustre-OST0000 complete [ 6553.056469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6553.060152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6553.066708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6553.069581] Lustre: Skipped 1 previous similar message [ 6558.336107] LustreError: 282769:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6558.338194] LustreError: 282769:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6558.365556] Lustre: server umount lustre-MDT0000 complete [ 6559.200061] LustreError: 279527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6559.206938] LustreError: 279527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6559.768867] LustreError: 279507:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516584 with bad export cookie 16118799689809002776 [ 6559.773197] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6559.774615] LustreError: 279507:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6559.849146] LustreError: 282971:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6559.851732] LustreError: 282971:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6559.911111] Lustre: server umount lustre-MDT0001 complete [ 6564.133161] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6565.268910] Key type lgssc unregistered [ 6565.407762] LNet: 283502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6565.411896] LNetError: 283502:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6565.419668] LNet: Removed LNI 192.168.204.115@tcp [ 6565.745158] Key type .llcrypt unregistered [ 6565.746806] Key type ._llcrypt unregistered [ 6573.708053] Key type ._llcrypt registered [ 6573.709277] Key type .llcrypt registered [ 6573.755153] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6574.098467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6574.159615] alg: No test for adler32 (adler32-zlib) [ 6575.048266] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 6575.150675] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6576.743237] Key type lgssc registered [ 6577.206707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6580.265125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6580.272841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6581.501791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6583.020322] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6586.276592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6586.425729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6587.943973] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6590.096574] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6591.547651] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6594.197576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6594.334202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6596.560028] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6598.571522] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6607.661492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6617.056789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6617.060527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6617.066429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6618.084690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6618.085697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6618.085882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6619.311135] LustreError: 287130:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6619.365778] Lustre: server umount lustre-OST0000 complete [ 6622.177384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6622.179910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6622.186401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6622.188935] Lustre: Skipped 1 previous similar message [ 6627.006765] LustreError: 287331:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6627.008920] LustreError: 287331:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6627.037095] Lustre: server umount lustre-MDT0000 complete [ 6628.320049] LustreError: 284709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6628.326780] LustreError: 284709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6628.664636] LustreError: 286340:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516653 with bad export cookie 4469830577953617833 [ 6628.669531] LustreError: 286340:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6628.670414] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6628.742356] LustreError: 287533:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6628.744980] LustreError: 287533:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6628.813136] Lustre: server umount lustre-MDT0001 complete [ 6633.132289] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6634.445690] Key type lgssc unregistered [ 6634.603603] LNet: 288065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6634.607375] LNetError: 288065:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6634.617377] LNet: Removed LNI 192.168.204.115@tcp [ 6635.012520] Key type .llcrypt unregistered [ 6635.014329] Key type ._llcrypt unregistered [ 6639.534258] Key type ._llcrypt registered [ 6639.535996] Key type .llcrypt registered [ 6639.582905] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6646.071534] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6652.488906] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6652.808315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6652.836334] alg: No test for adler32 (adler32-zlib) [ 6653.719155] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 6653.816486] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6655.407270] Key type lgssc registered [ 6655.829566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6659.155725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6661.548155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6663.710780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6666.084386] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6668.623079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6668.649232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6668.657947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6669.752375] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6669.765621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6669.805061] Lustre: lustre-MDT0000: new disk, initializing [ 6669.836104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6669.843963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6671.296620] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6675.791592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6675.820759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6675.865359] Lustre: 291548:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6675.894637] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6675.897149] Lustre: Skipped 1 previous similar message [ 6675.936315] Lustre: lustre-MDT0001: new disk, initializing [ 6675.977785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6675.996096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6676.000848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6677.444392] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6680.735760] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6682.014821] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6684.408255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6684.436568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6684.538070] Lustre: lustre-OST0000: new disk, initializing [ 6684.540260] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6684.571503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6686.017099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6686.021811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6686.060326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6686.574019] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6689.831789] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6691.292378] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6691.388111] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6692.833598] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6692.907419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6696.418668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6696.419536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6696.423225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6696.427569] Lustre: Skipped 1 previous similar message [ 6699.629117] LustreError: 294484:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6699.670012] Lustre: server umount lustre-OST0000 complete [ 6701.536905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6701.537218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6701.537825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6701.537831] Lustre: Skipped 1 previous similar message [ 6701.540044] LustreError: Skipped 1 previous similar message [ 6701.544641] Lustre: Skipped 1 previous similar message [ 6706.656396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6706.658807] Lustre: Skipped 2 previous similar messages [ 6711.777530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6711.779936] Lustre: Skipped 1 previous similar message [ 6715.359120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6715.399417] LustreError: 294686:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6715.401940] LustreError: 294686:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6715.429215] Lustre: server umount lustre-MDT0000 complete [ 6716.664165] LustreError: 292328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516741 with bad export cookie 12879856128177714200 [ 6716.667147] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6716.668091] LustreError: 292328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6716.730922] LustreError: 294890:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6716.733726] LustreError: 294890:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6716.781826] Lustre: server umount lustre-MDT0001 complete [ 6719.664557] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 20:45:45 (1763516745) [ 6722.021057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6722.213086] LustreError: 295560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6722.221067] LustreError: 295560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6722.242106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6723.676395] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6726.749367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6726.874395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6728.171979] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6729.938736] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6731.145796] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6733.479708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6735.498646] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6737.409502] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6752.224704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6752.226570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6752.227597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6752.235402] Lustre: Skipped 1 previous similar message [ 6756.846101] LustreError: 297830:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6756.849030] LustreError: 297830:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6756.890925] Lustre: server umount lustre-OST0000 complete [ 6760.929452] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6760.933093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6760.939610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6760.942105] Lustre: Skipped 1 previous similar message [ 6764.420637] LustreError: 298033:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6764.424036] LustreError: 298033:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6764.471156] Lustre: server umount lustre-MDT0000 complete [ 6765.998176] LustreError: 297178:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516791 with bad export cookie 12879856128177715068 [ 6765.999196] LustreError: 296281:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 6766.000653] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6766.003642] LustreError: 297178:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6766.164811] Lustre: server umount lustre-MDT0001 complete [ 6769.973172] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6771.034380] Key type lgssc unregistered [ 6771.166445] LNet: 298765:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6771.170114] LNetError: 298765:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6771.181400] LNet: Removed LNI 192.168.204.115@tcp [ 6771.525527] Key type .llcrypt unregistered [ 6771.527159] Key type ._llcrypt unregistered [ 6781.033036] Key type ._llcrypt registered [ 6781.034191] Key type .llcrypt registered [ 6781.072525] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6787.342856] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6787.769954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6787.779060] alg: No test for adler32 (adler32-zlib) [ 6788.633971] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 6788.718332] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6790.303147] Key type lgssc registered [ 6790.662639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6793.591569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6795.730060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6797.724159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6799.910981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6801.552557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6803.240384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6805.567145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6805.586290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6805.591798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6806.670454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6806.681850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6806.713280] Lustre: lustre-MDT0000: new disk, initializing [ 6806.736094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6806.743521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6808.034212] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6812.184249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6812.209405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6812.244384] Lustre: 302541:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6812.271458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6812.274052] Lustre: Skipped 1 previous similar message [ 6812.313727] Lustre: lustre-MDT0001: new disk, initializing [ 6812.348758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6812.362724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6812.366560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6813.640059] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6816.840179] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6818.164501] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6820.648339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6820.681594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6820.786397] Lustre: lustre-OST2710: new disk, initializing [ 6820.789353] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6820.825717] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6822.830715] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6824.377693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6824.381931] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6824.418458] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6826.191712] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 6828.752248] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6828.784348] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6828.848937] Lustre: lustre-OST03e8: new disk, initializing [ 6828.851704] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6828.892209] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6829.713593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6829.718626] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6829.765444] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6831.093986] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6833.091143] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 6835.999187] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6836.091121] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6837.655491] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6837.748214] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6839.263088] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6839.339768] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6840.802780] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6840.885104] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6843.360145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6843.363215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6843.369236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6845.414170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6845.414188] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6845.417090] Lustre: Skipped 1 previous similar message [ 6845.421400] Lustre: Skipped 2 previous similar messages [ 6848.128480] LustreError: 306880:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6848.165154] Lustre: server umount lustre-MDT0000 complete [ 6849.419175] LustreError: 302531:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516874 with bad export cookie 5734744433448347571 [ 6849.422015] LustreError: 302531:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6849.422080] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6849.482070] LustreError: 307082:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6849.483672] LustreError: 307082:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6849.529295] Lustre: server umount lustre-MDT0001 complete [ 6859.954084] LustreError: 307282:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6859.955834] LustreError: 307282:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6859.970932] Lustre: server umount lustre-OST2710 complete [ 6872.113192] LustreError: 307484:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6872.115558] LustreError: 307484:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6872.155227] Lustre: server umount lustre-OST03e8 complete [ 6878.064129] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6881.239087] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6884.360289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6886.410204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6888.435833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6890.496636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6892.663950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6892.685148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6892.758479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6892.768628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6892.804285] Lustre: lustre-MDT0000: new disk, initializing [ 6892.827944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6892.834755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6894.086252] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6898.119721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6898.140083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6898.173915] Lustre: 310274:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6898.275851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6899.579732] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6902.649492] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6903.725536] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6905.920448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6905.945546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6906.040667] Lustre: lustre-OST0000: new disk, initializing [ 6906.042403] Lustre: Skipped 1 previous similar message [ 6906.044281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6906.046894] Lustre: Skipped 2 previous similar messages [ 6906.072851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6906.075341] Lustre: Skipped 1 previous similar message [ 6906.212423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6906.215135] Lustre: Skipped 1 previous similar message [ 6906.216575] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6906.252854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6907.940599] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6911.006664] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6912.320073] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6912.396364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6913.636478] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6913.703665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6916.578170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6916.578333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6916.581442] LustreError: Skipped 1 previous similar message [ 6916.582039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6916.586126] Lustre: Skipped 1 previous similar message [ 6916.588015] Lustre: Skipped 1 previous similar message [ 6920.363066] LustreError: 313212:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6920.365071] LustreError: 313212:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6920.399147] Lustre: server umount lustre-OST0000 complete [ 6921.696205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6921.696772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6921.700946] Lustre: Skipped 1 previous similar message [ 6921.702870] Lustre: Skipped 1 previous similar message [ 6926.816201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6926.818274] Lustre: Skipped 2 previous similar messages [ 6927.779346] Lustre: server umount lustre-MDT0000 complete [ 6929.014279] LustreError: 310267:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516954 with bad export cookie 5734744433448349237 [ 6929.016802] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6929.018706] LustreError: 310267:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6929.084303] LustreError: 313614:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6929.087061] LustreError: 313614:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6929.145290] Lustre: server umount lustre-MDT0001 complete [ 6932.181505] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 20:49:17 (1763516957) [ 6932.867112] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6933.579361] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 20:49:18 (1763516958) [ 6934.560102] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6937.149193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6938.449907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6939.698396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6941.025869] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6941.405123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6943.735832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6943.863881] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6943.873705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6943.941835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6945.283592] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6948.137116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6948.176241] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6948.187455] Lustre: 315739:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6949.631078] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6951.421937] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6952.641351] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6955.052872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6955.086026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6955.156427] Lustre: Denying initial registration attempt from nid 192.168.204.115@tcp, specified as failover [ 6955.159774] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6955.164923] LustreError: 317355:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 6955.167550] LustreError: 317355:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 6955.170304] LustreError: 317355:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6955.196227] Lustre: server umount lustre-OST0000 complete [ 6955.198065] LustreError: 317355:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6958.559916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6958.562560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6958.566562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6963.714083] LustreError: 317723:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6963.716510] LustreError: 317723:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6963.741427] Lustre: server umount lustre-MDT0000 complete [ 6965.005134] LustreError: 315731:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516990 with bad export cookie 5734744433448350105 [ 6965.008137] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6965.008200] LustreError: 315731:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6965.124695] Lustre: server umount lustre-MDT0001 complete [ 6968.671481] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6969.668253] Key type lgssc unregistered [ 6969.793353] LNet: 318456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6969.795414] LNetError: 318456:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6969.803278] LNet: Removed LNI 192.168.204.115@tcp [ 6970.068198] Key type .llcrypt unregistered [ 6970.069666] Key type ._llcrypt unregistered [ 6973.332364] Key type ._llcrypt registered [ 6973.333801] Key type .llcrypt registered [ 6973.371149] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6973.801957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6973.809741] alg: No test for adler32 (adler32-zlib) [ 6974.662558] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 6974.744201] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6976.327171] Key type lgssc registered [ 6976.676767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6978.886673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6980.140027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6981.352365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6982.585707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6982.924307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6989.327678] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6992.416169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6992.422401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6993.512146] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6993.520578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6993.580513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6994.799395] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6997.645962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6997.676217] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6997.688567] Lustre: 321185:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6997.783579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6999.135786] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7000.976344] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7002.215504] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7004.581369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7004.607985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7004.674418] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7004.734019] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7006.571863] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7008.294065] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7011.810796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7011.811310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7011.811730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7011.814418] Lustre: Skipped 1 previous similar message [ 7011.816616] LustreError: Skipped 1 previous similar message [ 7015.084078] LustreError: 323470:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7015.119815] Lustre: server umount lustre-OST0000 complete [ 7016.928934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7016.929760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7016.932091] Lustre: Skipped 1 previous similar message [ 7016.933430] Lustre: Skipped 1 previous similar message [ 7022.048651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7022.051344] Lustre: Skipped 2 previous similar messages [ 7022.523978] LustreError: 323671:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7022.526092] LustreError: 323671:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7022.545914] Lustre: server umount lustre-MDT0000 complete [ 7023.780059] LustreError: 321914:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517048 with bad export cookie 2520660556783639111 [ 7023.783172] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7023.783241] LustreError: 321914:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7023.842741] LustreError: 323873:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7023.845674] LustreError: 323873:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7023.898299] Lustre: server umount lustre-MDT0001 complete [ 7027.433777] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7028.444200] Key type lgssc unregistered [ 7028.569363] LNet: 324403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7028.571873] LNetError: 324403:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7028.580291] LNet: Removed LNI 192.168.204.115@tcp [ 7028.855747] Key type .llcrypt unregistered [ 7028.856666] Key type ._llcrypt unregistered [ 7037.730409] Key type ._llcrypt registered [ 7037.731577] Key type .llcrypt registered [ 7037.769828] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7038.132738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7038.166021] alg: No test for adler32 (adler32-zlib) [ 7039.021514] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 7039.104216] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7040.687162] Key type lgssc registered [ 7041.039575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7043.480373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7043.486654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7044.633916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7045.911182] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7048.699269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7048.816434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7050.029608] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7051.822817] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7053.049321] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7055.286550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7055.392638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7057.252942] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7059.061591] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7074.272046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7074.275592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7074.280330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7074.787572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7074.789106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7074.791472] Lustre: Skipped 1 previous similar message [ 7078.076871] LustreError: 328046:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7078.105400] Lustre: server umount lustre-MDT0000 complete [ 7079.317273] LustreError: 325759:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517104 with bad export cookie 3723446650337750788 [ 7079.321431] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7079.322371] LustreError: 325759:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7079.903899] LustreError: 325772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7079.904236] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7079.909479] LustreError: 325772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7079.915270] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7079.918906] Lustre: Skipped 2 previous similar messages [ 7085.023897] LustreError: 325773:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7085.027407] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7085.031640] LustreError: 325773:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7085.034063] Lustre: Skipped 1 previous similar message [ 7085.438410] LustreError: 328248:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7085.440370] LustreError: 328248:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7085.490102] Lustre: server umount lustre-MDT0001 complete [ 7087.599546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7090.272122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7104.479628] LustreError: 325088:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009fcd9924 x1849181668923392/t0(0) o250->MGC192.168.204.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7104.604438] LustreError: 327385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7104.609517] LustreError: 327385:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7104.633472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7104.655494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 7105.944841] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7108.811939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7108.970663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7109.988269] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7110.328595] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7112.179672] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7114.083694] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7116.835194] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 7118.017302] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7118.567206] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7120.352495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7120.356126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7120.361244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7120.362902] Lustre: Skipped 1 previous similar message [ 7124.449573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7124.453353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7129.569937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7129.572659] Lustre: Skipped 3 previous similar messages [ 7134.175118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7134.191719] LustreError: 330555:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7134.193826] LustreError: 330555:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7134.227093] Lustre: server umount lustre-OST0000 complete [ 7139.808746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7139.808891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7139.812598] Lustre: Skipped 1 previous similar message [ 7141.565462] LustreError: 330757:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7141.568168] LustreError: 330757:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7141.591756] Lustre: server umount lustre-MDT0000 complete [ 7142.762060] LustreError: 328249:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517167 with bad export cookie 3723446650337757942 [ 7142.764663] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7142.766876] LustreError: 328249:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7142.873655] Lustre: server umount lustre-MDT0001 complete [ 7146.350741] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7147.418921] Key type lgssc unregistered [ 7147.550401] LNet: 331494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7147.552835] LNetError: 331494:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7147.558413] LNet: Removed LNI 192.168.204.115@tcp [ 7147.862072] Key type .llcrypt unregistered [ 7147.863256] Key type ._llcrypt unregistered [ 7156.895539] Key type ._llcrypt registered [ 7156.897147] Key type .llcrypt registered [ 7156.935471] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7157.293429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7157.325072] alg: No test for adler32 (adler32-zlib) [ 7158.176740] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 7158.260701] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7159.831188] Key type lgssc registered [ 7160.198449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7162.741949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7162.748569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7163.907499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7165.273891] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7168.085019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7168.199384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7169.464359] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7171.207926] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7172.413380] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7173.455709] LustreError: 332897:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7173.460261] LustreError: 332897:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7173.464157] LustreError: 332897:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 7173.600328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7173.600477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7173.601204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7173.610759] Lustre: Skipped 1 previous similar message [ 7178.721244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7178.724447] Lustre: Skipped 2 previous similar messages [ 7179.583044] LustreError: 334256:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7179.609609] Lustre: server umount lustre-MDT0000 complete [ 7180.863682] LustreError: 332848:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517206 with bad export cookie 13511194767923850064 [ 7180.868370] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7180.869344] LustreError: 332848:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7180.935771] LustreError: 334460:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7180.938810] LustreError: 334460:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7181.000578] Lustre: server umount lustre-MDT0001 complete [ 7186.182897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7186.297940] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7186.310219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7186.389398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7187.714777] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7190.408461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7190.440194] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7190.450046] Lustre: 335229:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7190.540865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7191.713473] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7193.375126] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7194.476414] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7196.997725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7197.063678] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7197.067894] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7197.070684] LustreError: 336886:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 7197.072818] LustreError: 336886:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 7197.074800] LustreError: 336886:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 7197.094330] Lustre: server umount lustre-OST0000 complete [ 7197.095573] LustreError: 336886:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7199.791367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7199.869249] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7199.940532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7201.769803] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7203.480715] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7205.978643] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7206.003230] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7206.027220] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7206.052285] Lustre: lustre-OST0001: new disk, initializing [ 7206.054145] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7206.194349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7206.197357] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7206.202218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 7206.230895] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7208.066189] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7209.787869] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7211.488604] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7211.488953] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7211.491595] LustreError: Skipped 1 previous similar message [ 7211.497639] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7211.499859] Lustre: Skipped 1 previous similar message [ 7212.781936] LustreError: 338999:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 7212.783699] LustreError: 338999:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7212.800989] Lustre: server umount lustre-OST0001 complete [ 7216.609665] LustreError: 338351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7216.610219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7216.614894] LustreError: 338351:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7216.618777] Lustre: Skipped 1 previous similar message [ 7216.622690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7216.623048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7216.628335] Lustre: Skipped 1 previous similar message [ 7221.731796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7221.731888] LustreError: 338351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7221.733323] Lustre: Skipped 1 previous similar message [ 7221.737381] LustreError: 338351:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7226.849667] LustreError: 337354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7226.855037] LustreError: 337354:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7228.383163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7228.404076] LustreError: 339201:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7228.406357] LustreError: 339201:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7228.438184] Lustre: server umount lustre-OST0000 complete [ 7231.455902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7231.457761] LustreError: Skipped 1 previous similar message [ 7231.459102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7231.462348] Lustre: Skipped 1 previous similar message [ 7231.464152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7231.466680] Lustre: Skipped 2 previous similar messages [ 7235.709676] LustreError: 339403:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7235.711690] LustreError: 339403:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7235.735768] Lustre: server umount lustre-MDT0000 complete [ 7236.881146] LustreError: 335222:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517262 with bad export cookie 13511194767923850596 [ 7236.884361] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7236.886017] LustreError: 335222:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7236.993336] Lustre: server umount lustre-MDT0001 complete [ 7240.244789] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7241.159202] Key type lgssc unregistered [ 7241.282357] LNet: 340135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7241.285277] LNetError: 340135:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7241.293276] LNet: Removed LNI 192.168.204.115@tcp [ 7241.581529] Key type .llcrypt unregistered [ 7241.582532] Key type ._llcrypt unregistered [ 7252.250924] Key type ._llcrypt registered [ 7252.252634] Key type .llcrypt registered [ 7252.291598] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 7258.781545] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7259.069480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7259.079182] alg: No test for adler32 (adler32-zlib) [ 7259.928907] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 7260.013622] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7261.599212] Key type lgssc registered [ 7261.972524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7265.088586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7267.382487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7269.602213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7271.740313] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7274.162633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7274.186241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7274.191695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7275.281726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7275.294376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7275.341745] Lustre: lustre-MDT0000: new disk, initializing [ 7275.366295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7275.374530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7276.698516] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7281.014941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7281.040280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7281.074495] Lustre: 344060:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7281.106610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7281.109412] Lustre: Skipped 1 previous similar message [ 7281.157264] Lustre: lustre-MDT0001: new disk, initializing [ 7281.194791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7281.208714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7281.211890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7282.657974] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7285.842921] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7287.029904] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7289.199679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7289.223057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7289.320778] Lustre: lustre-OST0000: new disk, initializing [ 7289.322607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7289.347940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7291.065605] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7294.030240] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7295.276834] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7296.499219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7296.502021] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7296.526231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7297.380085] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 7298.559158] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7298.623171] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7301.601507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7301.602370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7301.603776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7301.608787] Lustre: Skipped 1 previous similar message [ 7305.325115] LustreError: 347012:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7305.366959] Lustre: server umount lustre-OST0000 complete [ 7306.720372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7306.720434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7306.720976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7306.720980] Lustre: Skipped 1 previous similar message [ 7306.722768] LustreError: Skipped 1 previous similar message [ 7306.730649] Lustre: Skipped 1 previous similar message [ 7311.840749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7311.842613] Lustre: Skipped 2 previous similar messages [ 7312.703433] LustreError: 347213:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7312.706188] LustreError: 347213:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7312.734225] Lustre: server umount lustre-MDT0000 complete [ 7313.966048] LustreError: 344051:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517339 with bad export cookie 10701577018907260848 [ 7313.969065] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7313.969099] LustreError: 344051:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7314.034263] LustreError: 347415:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7314.037428] LustreError: 347415:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7314.092494] Lustre: server umount lustre-MDT0001 complete [ 7316.793262] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 20:55:42 (1763517342) [ 7318.512933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7320.431685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7327.780498] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 7331.026628] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7334.323396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7336.498029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7338.473954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7340.588936] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7342.788034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7342.809035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7342.880897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7342.891864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7342.924682] Lustre: lustre-MDT0000: new disk, initializing [ 7342.946132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7342.951985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7344.110810] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7348.182011] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7348.205445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7348.245781] Lustre: 351183:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7348.267984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7348.270909] Lustre: Skipped 1 previous similar message [ 7348.306252] Lustre: lustre-MDT0001: new disk, initializing [ 7348.335634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7348.347463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7348.351553] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7349.615102] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7352.657964] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7353.833381] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7355.970862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7355.999487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7357.798819] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7359.091359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7359.094140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7359.124511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7360.802612] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7362.127649] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7362.198701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7363.413493] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7363.478308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7364.579388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7364.579562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7364.581560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7364.582738] LustreError: Skipped 1 previous similar message [ 7364.590564] Lustre: Skipped 1 previous similar message [ 7369.700676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7369.702310] Lustre: Skipped 2 previous similar messages [ 7370.158074] LustreError: 354119:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7370.161368] LustreError: 354119:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7370.206610] Lustre: server umount lustre-OST0000 complete [ 7374.303954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7374.306363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7377.472433] LustreError: 354320:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7377.476119] LustreError: 354320:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7377.508807] Lustre: server umount lustre-MDT0000 complete [ 7378.789167] LustreError: 351174:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517403 with bad export cookie 10701577018907261702 [ 7378.791921] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7378.792476] LustreError: 351174:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7378.916792] Lustre: server umount lustre-MDT0001 complete [ 7381.818231] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 20:56:47 (1763517407) [ 7384.349718] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 20:56:49 (1763517409) [ 7387.243588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7387.428584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7387.430991] Lustre: Skipped 1 previous similar message [ 7388.751196] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7391.729426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7393.232186] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7395.136546] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7396.400125] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7398.758845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7400.958474] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7402.894255] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7416.965802] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7417.534311] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7418.102417] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7418.656296] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7419.284239] Lustre: Failing over lustre-MDT0000 [ 7419.410393] LustreError: 357996:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7419.412167] LustreError: 357996:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7419.435995] Lustre: server umount lustre-MDT0000 complete [ 7422.433089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7422.434797] LustreError: 355528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7422.437664] Lustre: Skipped 3 previous similar messages [ 7422.437934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7422.442221] LustreError: 355528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7423.659163] LustreError: 355528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7427.552736] LustreError: 355534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7427.559670] LustreError: 355534:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7432.175960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7432.210989] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7432.296468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7432.298906] Lustre: Skipped 2 previous similar messages [ 7432.322165] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7433.523242] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7433.892616] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7437.796263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7437.803804] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7437.818391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7438.945863] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7439.482663] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7441.229686] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7441.867225] Lustre: Failing over lustre-MDT0000 [ 7441.998303] LustreError: 359072:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7442.000154] LustreError: 359072:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7442.025101] Lustre: server umount lustre-MDT0000 complete [ 7442.913383] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7442.914195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7442.914907] LustreError: 355534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7442.914920] LustreError: 355534:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7442.917443] Lustre: Skipped 1 previous similar message [ 7448.032634] LustreError: 355534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7448.040347] LustreError: 355534:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7455.049163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7455.088900] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7455.213137] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7456.587415] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7459.440371] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7460.327775] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7460.329726] Lustre: Skipped 2 previous similar messages [ 7460.334487] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7460.352491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7461.569769] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7462.204666] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7465.441330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7465.441900] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7465.443153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7465.443170] Lustre: Skipped 2 previous similar messages [ 7465.450323] Lustre: Skipped 2 previous similar messages [ 7470.685947] Lustre: server umount lustre-MDT0000 complete [ 7472.018234] LustreError: 357997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517497 with bad export cookie 10701577018907264369 [ 7472.021488] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7472.023322] LustreError: 357997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7472.134522] Lustre: server umount lustre-MDT0001 complete [ 7483.697953] LustreError: 360505:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7483.701138] LustreError: 360505:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7483.750358] Lustre: server umount lustre-OST0000 complete [ 7488.788050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7488.927204] LustreError: 361332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7488.934060] LustreError: 361332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7488.950855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7488.952837] Lustre: Skipped 1 previous similar message [ 7490.270636] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7493.169180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7494.593627] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7496.445182] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7497.730767] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7500.103891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7502.149056] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7503.919505] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7514.537039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7523.807815] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7523.809846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7523.813555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7523.815666] Lustre: Skipped 5 previous similar messages [ 7526.744528] Lustre: server umount lustre-OST0000 complete [ 7534.167424] Lustre: server umount lustre-MDT0000 complete [ 7535.073060] LustreError: 361328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7535.078472] LustreError: 361328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7535.510395] LustreError: 362950:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517560 with bad export cookie 10701577018907265076 [ 7535.513868] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7535.515219] LustreError: 362950:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7535.653287] Lustre: server umount lustre-MDT0001 complete [ 7539.317296] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7540.387950] Key type lgssc unregistered [ 7540.520361] LNet: 364535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7540.522667] LNetError: 364535:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7540.528411] LNet: Removed LNI 192.168.204.115@tcp [ 7540.840845] Key type .llcrypt unregistered [ 7540.842881] Key type ._llcrypt unregistered [ 7550.831150] Key type ._llcrypt registered [ 7550.832373] Key type .llcrypt registered [ 7550.867033] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7551.160867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7551.170950] alg: No test for adler32 (adler32-zlib) [ 7552.024878] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 7552.113892] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7553.703202] Key type lgssc registered [ 7554.072676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7556.800106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7556.806398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7557.976781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7559.306041] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7562.271615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7562.414971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7563.839575] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7565.702488] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7566.940840] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7569.345044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7569.472928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7571.473185] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7573.302097] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7583.843378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7588.320272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7588.323429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7588.328851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7589.345303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7589.346020] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.918113] LustreError: 368276:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7593.945266] Lustre: server umount lustre-MDT0000 complete [ 7594.464541] LustreError: 366009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7594.470976] LustreError: 366009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7595.242488] LustreError: 368277:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517620 with bad export cookie 8648411620806959086 [ 7595.247279] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7595.248619] LustreError: 368277:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7595.316977] LustreError: 368478:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7595.318945] LustreError: 368478:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7595.372653] Lustre: server umount lustre-MDT0001 complete [ 7605.937295] LustreError: 368679:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7605.939317] LustreError: 368679:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7605.977552] Lustre: server umount lustre-OST0000 complete [ 7610.175544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7610.306541] LustreError: 369409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7610.325645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7611.482606] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7614.226161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7614.333754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7615.543458] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7617.259809] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7618.520532] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7620.714170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7622.680708] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7624.459755] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7635.173720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7650.272044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7650.275691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7650.281361] Lustre: Skipped 1 previous similar message [ 7650.283979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7650.285473] Lustre: Skipped 2 previous similar messages [ 7653.113558] LustreError: 371793:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7653.115408] LustreError: 371793:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7653.138363] Lustre: server umount lustre-MDT0000 complete [ 7654.350905] LustreError: 369389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517679 with bad export cookie 8648411620806960185 [ 7654.356145] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7654.356428] LustreError: 369389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7654.479425] Lustre: server umount lustre-MDT0001 complete [ 7664.946804] LustreError: 372196:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7664.948652] LustreError: 372196:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7664.995560] Lustre: server umount lustre-OST0000 complete [ 7670.067966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7670.222182] LustreError: 373023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7670.250336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7670.253199] Lustre: Skipped 1 previous similar message [ 7671.469251] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7674.306662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7675.615560] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7677.239114] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7678.420659] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7680.775109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7682.908788] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7684.752313] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7695.206481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7699.424017] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7699.426106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7699.429092] Lustre: Skipped 2 previous similar messages [ 7699.430560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7699.432211] Lustre: Skipped 2 previous similar messages [ 7702.764097] LustreError: 375288:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7702.767368] LustreError: 375288:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7702.804387] Lustre: server umount lustre-OST0000 complete [ 7705.056088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7705.058260] LustreError: Skipped 1 previous similar message [ 7705.059548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7705.062734] Lustre: Skipped 1 previous similar message [ 7705.064430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7705.067756] Lustre: Skipped 2 previous similar messages [ 7715.808065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7715.809909] Lustre: Skipped 4 previous similar messages [ 7718.367127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7718.427498] Lustre: server umount lustre-MDT0000 complete [ 7719.654160] LustreError: 373003:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517744 with bad export cookie 8648411620806961193 [ 7719.657392] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7719.659536] LustreError: 373003:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7719.718858] LustreError: 375693:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7719.720901] LustreError: 375693:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 7719.766574] Lustre: server umount lustre-MDT0001 complete [ 7723.407932] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7724.518211] Key type lgssc unregistered [ 7724.641315] LNet: 376224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7724.643948] LNetError: 376224:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7724.653354] LNet: Removed LNI 192.168.204.115@tcp [ 7724.967220] Key type .llcrypt unregistered [ 7724.968674] Key type ._llcrypt unregistered [ 7734.086573] Key type ._llcrypt registered [ 7734.087608] Key type .llcrypt registered [ 7734.124184] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7734.464593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7734.515082] alg: No test for adler32 (adler32-zlib) [ 7735.381652] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 7735.469581] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7737.055215] Key type lgssc registered [ 7737.441726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7740.120341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7740.125798] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7740.128331] LustreError: 377620:0:(osd_handler.c:8577:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7740.135915] LustreError: 377620:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7740.138093] LustreError: 377620:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7740.140181] LustreError: 377620:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7740.142677] LustreError: 377620:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7742.722303] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7742.726423] LustreError: 378020:0:(osd_handler.c:8577:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7742.736133] LustreError: 378020:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7742.739556] LustreError: 378020:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7742.741676] LustreError: 378020:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7742.744208] LustreError: 378020:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7749.281303] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7750.355221] Key type lgssc unregistered [ 7750.483324] LNet: 379004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7750.485671] LNetError: 379004:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7750.494268] LNet: Removed LNI 192.168.204.115@tcp [ 7750.801457] Key type .llcrypt unregistered [ 7750.802990] Key type ._llcrypt unregistered [ 7757.031707] Key type ._llcrypt registered [ 7757.033363] Key type .llcrypt registered [ 7757.079533] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 7763.114365] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7763.415241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7763.446033] alg: No test for adler32 (adler32-zlib) [ 7764.291180] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 7764.367700] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7765.943195] Key type lgssc registered [ 7766.343397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7769.114882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7771.200224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7773.195181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7775.265571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7777.492682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7777.511203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7777.515854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7778.605923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7778.619085] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7778.658851] Lustre: lustre-MDT0000: new disk, initializing [ 7778.684982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7778.691274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7779.916243] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7783.922276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7783.939174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7783.974150] Lustre: 382109:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7783.996045] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7783.997934] Lustre: Skipped 1 previous similar message [ 7784.032937] Lustre: lustre-MDT0001: new disk, initializing [ 7784.063597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7784.074269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7784.078236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7785.340982] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7788.506127] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7789.679329] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7791.817832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7791.838391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7791.927166] Lustre: lustre-OST0000: new disk, initializing [ 7791.929734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7791.956876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7793.773922] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7796.733858] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7797.748581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7797.751332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7797.783198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7798.028645] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7798.094566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7799.358342] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7799.427458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7802.850520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7802.850701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7802.851806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7802.854837] Lustre: Skipped 1 previous similar message [ 7802.859854] LustreError: Skipped 1 previous similar message [ 7806.126089] LustreError: 385050:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7806.164989] Lustre: server umount lustre-OST0000 complete [ 7807.968389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7807.968915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7807.974360] Lustre: Skipped 1 previous similar message [ 7807.975942] Lustre: Skipped 1 previous similar message [ 7813.088320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7813.090253] Lustre: Skipped 2 previous similar messages [ 7813.569903] LustreError: 385251:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7813.571855] LustreError: 385251:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7813.602140] Lustre: server umount lustre-MDT0000 complete [ 7814.942128] LustreError: 382101:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517840 with bad export cookie 3335735990249829475 [ 7814.945282] LustreError: 382101:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7814.945602] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7815.012327] LustreError: 385453:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7815.015176] LustreError: 385453:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7815.066335] Lustre: server umount lustre-MDT0001 complete [ 7818.043149] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7818.633456] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 21:04:03 (1763517843) [ 7821.148553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7821.346032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7822.907306] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7825.921228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7826.055209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7827.461385] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7829.473423] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7830.772970] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7833.146185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7835.477023] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7837.548539] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7840.024611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7840.049363] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7840.110340] Lustre: lustre-OST0001: new disk, initializing [ 7840.112272] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7840.139755] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7840.142103] Lustre: Skipped 1 previous similar message [ 7841.088918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7841.097460] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7841.126300] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7841.926834] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7844.843076] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7851.182841] Lustre: lustre-OST0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 7851.491108] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7851.493673] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7851.494635] Lustre: Skipped 1 previous similar message [ 7851.496592] LustreError: Skipped 1 previous similar message [ 7852.780073] LustreError: 389368:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7852.781976] LustreError: 389368:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7852.798813] Lustre: server umount lustre-OST0001 complete [ 7856.613317] LustreError: 387765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7856.618911] LustreError: 387765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7861.732166] LustreError: 387766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7861.738030] LustreError: 387766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7866.850639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7866.851835] LustreError: 387765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7866.851935] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7866.854903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7866.861713] LustreError: 387765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7866.869739] Lustre: Skipped 3 previous similar messages [ 7871.970887] LustreError: 387767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7871.976244] LustreError: 387767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7877.088413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7877.090650] Lustre: Skipped 2 previous similar messages [ 7877.091825] LustreError: 387766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7877.099539] LustreError: 387766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7879.135133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7879.152091] LustreError: 389590:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7879.154529] LustreError: 389590:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7879.191578] Lustre: server umount lustre-OST0000 complete [ 7882.209230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7882.215512] Lustre: Skipped 1 previous similar message [ 7886.811791] Lustre: server umount lustre-MDT0000 complete [ 7887.328386] LustreError: 386171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7887.335888] LustreError: 386171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7888.151244] LustreError: 388673:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517913 with bad export cookie 3335735990249830343 [ 7888.153977] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7888.155330] LustreError: 388673:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7894.271399] LustreError: 389993:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7894.274195] LustreError: 389993:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7894.334604] Lustre: server umount lustre-MDT0001 complete [ 7898.175808] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7899.284175] Key type lgssc unregistered [ 7899.442340] LNet: 390526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7899.444527] LNetError: 390526:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7899.455339] LNet: Removed LNI 192.168.204.115@tcp [ 7899.789240] Key type .llcrypt unregistered [ 7899.790723] Key type ._llcrypt unregistered [ 7911.921973] Key type ._llcrypt registered [ 7911.923152] Key type .llcrypt registered [ 7911.962097] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 7917.805644] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7918.245047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7918.257064] alg: No test for adler32 (adler32-zlib) [ 7919.109058] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 7919.184933] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7920.767101] Key type lgssc registered [ 7921.083407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7924.142374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7926.270350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7928.440566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7930.629676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7932.905652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7932.931448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7932.936168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7934.020661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7934.032130] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7934.071926] Lustre: lustre-MDT0000: new disk, initializing [ 7934.095868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7934.102205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7935.384771] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7939.734298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7939.757577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7939.794877] Lustre: 394447:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7939.818660] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7939.821051] Lustre: Skipped 1 previous similar message [ 7939.863648] Lustre: lustre-MDT0001: new disk, initializing [ 7939.899459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7939.913150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7939.918121] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7941.328786] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7944.512827] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7945.744524] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7948.055904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7948.083750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7948.175010] Lustre: lustre-OST0000: new disk, initializing [ 7948.177619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7948.205691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7950.072072] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7953.225491] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7954.717588] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7954.939772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7954.945431] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7954.976751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7955.809564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 7957.052369] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7957.120117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7960.032719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7960.032920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7960.035934] LustreError: Skipped 1 previous similar message [ 7960.041199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7960.044331] Lustre: Skipped 1 previous similar message [ 7963.824094] LustreError: 397388:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7963.861808] Lustre: server umount lustre-OST0000 complete [ 7965.152373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7965.153041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7965.157875] Lustre: Skipped 2 previous similar messages [ 7970.271924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7970.273558] Lustre: Skipped 2 previous similar messages [ 7971.195206] LustreError: 397591:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7971.197122] LustreError: 397591:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7971.223535] Lustre: server umount lustre-MDT0000 complete [ 7972.443138] LustreError: 394436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517997 with bad export cookie 12501646481385256134 [ 7972.446548] LustreError: 394436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7972.447135] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7972.528143] LustreError: 397793:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7972.530879] LustreError: 397793:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7972.588133] Lustre: server umount lustre-MDT0001 complete [ 7975.478969] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 21:06:40 (1763518000) [ 7978.548494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7981.531768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7981.681079] Lustre: lustre-MDT0000: new disk, initializing [ 7981.714813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7983.142123] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7986.146938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7986.275240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7987.593309] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7989.475369] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7990.796981] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7996.383881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7996.387593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7996.393887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7997.627990] LustreError: 400401:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7997.631644] LustreError: 400401:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7997.656433] Lustre: server umount lustre-MDT0000 complete [ 7997.920230] LustreError: 399076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7997.925427] LustreError: 399076:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7998.908640] LustreError: 399037:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763518024 with bad export cookie 12501646481385257002 [ 7998.912285] LustreError: 399037:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7998.913465] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7999.045144] Lustre: server umount lustre-MDT0001 complete [ 8002.269149] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 21:07:07 (1763518027) [ 8004.513799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8004.670349] LustreError: 401320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8004.693069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8006.126496] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8009.009354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8010.517359] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8012.287355] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8013.529408] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8015.853875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8017.913592] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8019.790773] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8033.156955] Lustre: Permanently deactivating lustre-OST0000 [ 8033.162595] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8042.485655] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8042.908262] LustreError: 404128:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8042.911933] LustreError: 404128:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 8042.914440] LustreError: 404128:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8043.537871] LustreError: 404176:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 8043.540761] LustreError: 404176:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8050.222074] LustreError: 404273:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8050.224128] LustreError: 404273:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8050.255656] Lustre: server umount lustre-OST0000 complete [ 8051.179580] LustreError: 404425:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 8051.182038] LustreError: 404425:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8055.264897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8055.265055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8055.265586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8055.265590] Lustre: Skipped 1 previous similar message [ 8055.269254] Lustre: Skipped 2 previous similar messages [ 8062.435076] LustreError: 404522:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8062.437123] LustreError: 404522:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8062.465301] Lustre: server umount lustre-MDT0000 complete [ 8063.754134] LustreError: 403688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763518088 with bad export cookie 12501646481385257541 [ 8063.758502] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8063.758870] LustreError: 403688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8073.229108] Lustre: server umount lustre-MDT0001 complete [ 8076.269934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8077.623711] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8078.593265] LustreError: 405566:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 8080.749840] Lustre: 405904:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.115@tcp [ 8081.437517] LustreError: 406001:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8081.440614] LustreError: 406001:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8081.466992] Lustre: server umount MGS complete [ 8085.468327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8085.629448] LustreError: 406628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8085.635201] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8085.642248] Lustre: Skipped 1 previous similar message [ 8085.664807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8085.667386] Lustre: Skipped 2 previous similar messages [ 8087.046628] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8090.031469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8090.128053] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8091.615835] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8093.612664] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8094.881579] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8095.522617] Lustre: Permanently reactivating lustre-OST0000 [ 8095.528128] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8095.529814] Lustre: Skipped 2 previous similar messages [ 8124.895170] LustreError: 408326:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 8127.747283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8127.875123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8127.878532] Lustre: Skipped 1 previous similar message [ 8129.849158] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8131.791140] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8144.351869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8144.355619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8144.362147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8144.364625] Lustre: Skipped 3 previous similar messages [ 8145.965924] LustreError: 410154:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8145.968529] LustreError: 410154:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 8146.001502] Lustre: server umount lustre-OST0000 complete [ 8149.401662] Lustre: server umount lustre-MDT0000 complete [ 8150.497447] LustreError: 406628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8150.502595] LustreError: 406628:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8150.727432] LustreError: 406609:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763518175 with bad export cookie 12501646481385258486 [ 8150.730847] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8150.733156] LustreError: 406609:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8150.854445] Lustre: server umount lustre-MDT0001 complete [ 8154.919292] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8156.021816] Key type lgssc unregistered [ 8156.153395] LNet: 411088:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8156.156536] LNetError: 411088:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8156.163363] LNet: Removed LNI 192.168.204.115@tcp [ 8156.473434] Key type .llcrypt unregistered [ 8156.474839] Key type ._llcrypt unregistered [ 8162.713417] Key type ._llcrypt registered [ 8162.715501] Key type .llcrypt registered [ 8162.756069] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 8169.164436] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8169.538865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8169.570053] alg: No test for adler32 (adler32-zlib) [ 8170.442407] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 8170.543082] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8172.127132] Key type lgssc registered [ 8172.547904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8175.661496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8177.799069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8179.918777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8182.083562] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8184.422589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8184.447915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8184.455134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8185.549331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8185.562835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8185.610199] Lustre: lustre-MDT0000: new disk, initializing [ 8185.639765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8185.648156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8187.050956] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8191.304116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8191.329597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8191.371757] Lustre: 414192:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8191.397061] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8191.398958] Lustre: Skipped 1 previous similar message [ 8191.434674] Lustre: lustre-MDT0001: new disk, initializing [ 8191.472306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8191.488155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8191.493506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8192.816978] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8195.983747] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8197.203807] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8199.436315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8199.465197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8199.560807] Lustre: lustre-OST0000: new disk, initializing [ 8199.562850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8199.589419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8201.547328] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8202.048890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8202.052738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8202.101417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8204.827417] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8206.256795] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8206.337703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8207.671252] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8207.743549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8212.452835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8212.454252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8212.456239] LustreError: Skipped 1 previous similar message [ 8212.463204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8212.466576] Lustre: Skipped 1 previous similar message [ 8214.446093] LustreError: 417131:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8214.486530] Lustre: server umount lustre-OST0000 complete [ 8217.055911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8217.058638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8217.062653] Lustre: Skipped 1 previous similar message [ 8217.064780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8222.688599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8222.691053] Lustre: Skipped 2 previous similar messages [ 8227.809399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8227.811386] Lustre: Skipped 1 previous similar message [ 8230.367179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8230.403681] LustreError: 417333:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8230.405429] LustreError: 417333:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8230.438490] Lustre: server umount lustre-MDT0000 complete [ 8231.671147] LustreError: 414973:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763518256 with bad export cookie 14203061128931420929 [ 8231.674282] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8231.674661] LustreError: 414973:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8231.737633] LustreError: 417536:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8231.740965] LustreError: 417536:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8231.799094] Lustre: server umount lustre-MDT0001 complete [ 8234.435524] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8234.984887] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 21:11:00 (1763518260) [ 8237.088602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8237.243424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8238.422202] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8241.295901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8241.408772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8242.727254] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8244.458842] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8245.610397] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8247.832968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8249.800465] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8251.649331] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8252.331175] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8263.407029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8266.719760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8266.722507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8266.726269] Lustre: Skipped 1 previous similar message [ 8266.727667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8270.893101] LustreError: 420715:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8270.895076] LustreError: 420715:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8270.933479] Lustre: server umount lustre-OST0000 complete [ 8272.351978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8272.355119] LustreError: Skipped 1 previous similar message [ 8272.356355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8272.359488] Lustre: Skipped 1 previous similar message [ 8277.984664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8277.987364] Lustre: Skipped 5 previous similar messages [ 8278.265735] LustreError: 420917:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8278.267508] LustreError: 420917:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8278.292226] Lustre: server umount lustre-MDT0000 complete [ 8279.455627] LustreError: 418237:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763518304 with bad export cookie 14203061128931421797 [ 8279.459774] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8279.459963] LustreError: 418237:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8279.577154] Lustre: server umount lustre-MDT0001 complete [ 8283.002815] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8283.885142] Key type lgssc unregistered [ 8284.007345] LNet: 421648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8284.009410] LNetError: 421648:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8284.016276] LNet: Removed LNI 192.168.204.115@tcp [ 8284.305661] Key type .llcrypt unregistered [ 8284.306625] Key type ._llcrypt unregistered [ 8291.641177] Key type ._llcrypt registered [ 8291.642341] Key type .llcrypt registered [ 8291.677890] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8291.789255] Key type .llcrypt unregistered [ 8291.790572] Key type ._llcrypt unregistered [ 8298.671500] Key type ._llcrypt registered [ 8298.672666] Key type .llcrypt registered [ 8298.714991] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8299.114529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8299.151522] alg: No test for adler32 (adler32-zlib) [ 8300.013592] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 8300.097977] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8301.679143] Key type lgssc registered [ 8301.999521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8304.456828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8304.461859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8305.616816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8306.924768] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8309.636956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8309.746277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8311.537632] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8313.229159] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8314.851766] LustreError: 423796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8314.857557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8314.858397] LustreError: 423796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8315.462457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8315.586803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8316.818453] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8325.087831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8325.091113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8325.096628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8326.114393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8326.115801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8326.117154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8329.964076] LustreError: 425853:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 8330.000739] Lustre: server umount lustre-OST0000 complete [ 8334.818061] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8334.822324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8334.829679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8334.831931] Lustre: Skipped 1 previous similar message [ 8337.410183] LustreError: 426054:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8337.413633] LustreError: 426054:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8337.449382] Lustre: server umount lustre-MDT0000 complete [ 8338.866242] LustreError: 423775:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763518364 with bad export cookie 3872359223085262224 [ 8338.869871] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8338.871255] LustreError: 423775:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8338.939243] LustreError: 426256:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8338.941296] LustreError: 426256:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8338.992109] Lustre: server umount lustre-MDT0001 complete [ 8342.642482] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8343.726432] Key type lgssc unregistered [ 8343.851424] LNet: 426788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8343.854660] LNetError: 426788:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8343.864403] LNet: Removed LNI 192.168.204.115@tcp [ 8344.145607] Key type .llcrypt unregistered [ 8344.146598] Key type ._llcrypt unregistered [ 8353.288078] Key type ._llcrypt registered [ 8353.289351] Key type .llcrypt registered [ 8353.322619] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8353.829968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8353.837578] alg: No test for adler32 (adler32-zlib) [ 8354.703198] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 8354.795061] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8356.391185] Key type lgssc registered [ 8356.804334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8359.311536] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8359.317648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8374.751321] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8379.871201] LustreError: 428134:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [ 8406.495177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8408.485632] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8410.406762] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8412.770716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8412.968094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8412.989718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8414.350695] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8417.379059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8417.521984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8418.955956] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8420.966033] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8422.288071] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8426.978471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8426.979965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8426.982752] Lustre: Skipped 1 previous similar message [ 8426.987763] Lustre: Skipped 1 previous similar message [ 8430.765973] LustreError: 430415:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 8430.805275] Lustre: server umount lustre-OST0000 complete [ 8433.119928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8433.123058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8433.127788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8437.216249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8437.216700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8438.271810] LustreError: 430617:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 8438.273982] LustreError: 430617:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8438.295811] Lustre: server umount lustre-MDT0000 complete [ 8439.583082] LustreError: 430329:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763518464 with bad export cookie 10440716659205985416 [ 8439.586073] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8439.586547] LustreError: 430329:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8439.658769] LustreError: 430819:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8439.661177] LustreError: 430819:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8439.714495] Lustre: server umount lustre-MDT0001 complete [ 8443.481207] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8444.568962] Key type lgssc unregistered [ 8444.701384] LNet: 431350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8444.704945] LNetError: 431350:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8444.712354] LNet: Removed LNI 192.168.204.115@tcp [ 8445.008582] Key type .llcrypt unregistered [ 8445.009966] Key type ._llcrypt unregistered [ 8454.601325] Key type ._llcrypt registered [ 8454.602879] Key type .llcrypt registered [ 8454.650476] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8455.138393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8455.144539] alg: No test for adler32 (adler32-zlib) [ 8456.001903] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 8456.084028] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8457.671132] Key type lgssc registered [ 8458.041578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8460.786660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8460.793023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8461.958731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8463.389558] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8466.345914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8466.473900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8467.823158] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8469.699638] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8470.961077] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8473.277363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8473.393761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8475.276942] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8477.152841] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8487.722206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8497.120304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8497.123426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8497.128210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8497.838148] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 8498.146164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8498.150032] Lustre: Skipped 1 previous similar message [ 8499.707632] LustreError: 435004:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8499.737664] Lustre: server umount lustre-MDT0000 complete [ 8500.506490] LustreError: 432718:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.15@tcp arrived at 1763518525 with bad export cookie 5252486111612646519 [ 8500.511960] LustreError: 432718:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8501.160161] LustreError: 432718:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763518526 with bad export cookie 5252486111612645868 [ 8501.164589] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8502.239988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8502.242570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8502.247169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8502.249332] Lustre: Skipped 3 previous similar messages [ 8503.263928] LustreError: 432737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8503.269214] LustreError: 432737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8506.272033] LustreError: 432737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8506.272444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8506.277146] LustreError: 432737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8506.281987] Lustre: Skipped 1 previous similar message [ 8507.244629] LustreError: 435207:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8507.246858] LustreError: 435207:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8507.285798] Lustre: server umount lustre-OST0000 complete [ 8509.548559] LustreError: 435562:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8509.550639] LustreError: 435562:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8509.605647] Lustre: server umount lustre-MDT0001 complete [ 8513.284181] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8514.308811] Key type lgssc unregistered [ 8514.432395] LNet: 436093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8514.434890] LNetError: 436093:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8514.444260] LNet: Removed LNI 192.168.204.115@tcp [ 8514.723492] Key type .llcrypt unregistered [ 8514.725232] Key type ._llcrypt unregistered [ 8523.743338] Key type ._llcrypt registered [ 8523.744566] Key type .llcrypt registered [ 8523.785033] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8524.219283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8524.281356] alg: No test for adler32 (adler32-zlib) [ 8525.158389] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 8525.257854] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8526.847095] Key type lgssc registered [ 8527.269414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8529.970707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8529.976967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8531.134875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8532.457294] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8535.216548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8535.353594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8536.728826] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8538.582342] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8539.804734] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8542.147364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8542.289877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8544.213578] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8546.025713] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8556.644230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8561.119813] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8561.122087] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8561.125983] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8561.838457] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 8562.146612] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8563.710352] LustreError: 439734:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8563.731113] Lustre: server umount lustre-MDT0001 complete [ 8567.264871] LustreError: 437467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8567.266230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8567.266663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8567.267280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8567.267284] Lustre: Skipped 2 previous similar messages [ 8567.270583] LustreError: 437467:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8571.631089] LustreError: 439939:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8571.633413] LustreError: 439939:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8571.672922] Lustre: server umount lustre-OST0000 complete [ 8572.385600] LustreError: 437467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8573.002700] LustreError: 440140:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8573.004659] LustreError: 440140:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8573.067283] Lustre: server umount lustre-MDT0000 complete [ 8577.855769] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8578.998550] Key type lgssc unregistered [ 8579.140527] LNet: 440823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8579.144381] LNetError: 440823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8579.153620] LNet: Removed LNI 192.168.204.115@tcp [ 8579.503322] Key type .llcrypt unregistered [ 8579.504904] Key type ._llcrypt unregistered [ 8587.950686] Key type ._llcrypt registered [ 8587.952109] Key type .llcrypt registered [ 8587.989791] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 8594.243531] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8594.613408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8594.667888] alg: No test for adler32 (adler32-zlib) [ 8595.520832] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 8595.603317] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8597.183170] Key type lgssc registered [ 8597.553458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8600.586760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8602.757244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8604.925628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8607.114899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8610.413861] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8613.441614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8615.832721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8615.855937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8615.861021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8616.945024] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8616.956286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8616.995750] Lustre: lustre-MDT0000: new disk, initializing [ 8617.020363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8617.027201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8618.360030] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8621.960329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8624.225284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8624.248818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8624.348679] Lustre: lustre-OST0000: new disk, initializing [ 8624.351655] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8624.354693] Lustre: Skipped 1 previous similar message [ 8624.383173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8626.416510] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8629.652078] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8632.830708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8632.835356] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8632.845351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8638.222021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8640.642614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8640.673693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8640.708985] Lustre: 444767:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8640.742818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8640.787579] Lustre: lustre-MDT0001: new disk, initializing [ 8640.840590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8640.856407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8640.859737] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8642.179661] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8645.780648] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8645.853208] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8647.175201] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8647.263483] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8652.257611] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8652.258469] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8652.260376] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8652.264179] Lustre: Skipped 1 previous similar message [ 8654.975352] LustreError: 448189:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8655.001527] Lustre: server umount lustre-MDT0001 complete [ 8662.498767] LustreError: 445414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8662.506590] LustreError: 445414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8667.616108] LustreError: 444777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8667.616391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8667.625245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8667.626794] Lustre: Skipped 2 previous similar messages [ 8672.736823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8672.737887] LustreError: 444778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8673.453246] LustreError: 448461:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 8673.455391] LustreError: 448461:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8673.490422] Lustre: server umount lustre-OST0000 complete [ 8681.023334] LustreError: 448662:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 8681.025181] LustreError: 448662:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8681.080178] Lustre: server umount lustre-MDT0000 complete [ 8685.622704] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8686.653850] Key type lgssc unregistered [ 8686.802450] LNet: 449347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8686.805442] LNetError: 449347:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8686.812444] LNet: Removed LNI 192.168.204.115@tcp [ 8687.155061] Key type .llcrypt unregistered [ 8687.156588] Key type ._llcrypt unregistered [ 8700.543897] Key type ._llcrypt registered [ 8700.544995] Key type .llcrypt registered [ 8700.580730] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8700.689297] Key type .llcrypt unregistered [ 8700.690326] Key type ._llcrypt unregistered [ 8707.421788] Key type ._llcrypt registered [ 8707.422942] Key type .llcrypt registered [ 8707.459207] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8707.773566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8707.806765] alg: No test for adler32 (adler32-zlib) [ 8708.670822] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 8708.763854] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8710.351169] Key type lgssc registered [ 8710.720097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8713.336406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8715.537452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8717.665970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8719.927204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8719.949889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8719.954165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8721.060853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8721.076714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8721.124359] Lustre: lustre-MDT0000: new disk, initializing [ 8721.150664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8721.156451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8722.557269] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8726.851026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8726.873897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8726.912302] Lustre: 452663:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8726.935596] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8726.937290] Lustre: Skipped 1 previous similar message [ 8726.978845] Lustre: lustre-MDT0001: new disk, initializing [ 8727.026613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8727.039868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8727.044227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8728.438894] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8731.604625] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8732.881764] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8735.184706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8735.209619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8735.339511] Lustre: lustre-OST0000: new disk, initializing [ 8735.344497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8735.350832] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 8735.378738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8737.313558] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8740.462225] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8740.840858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8740.843978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8740.856039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8751.074608] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8751.075696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8751.078704] Lustre: Skipped 2 previous similar messages [ 8751.080713] Lustre: Skipped 1 previous similar message [ 8753.922913] LustreError: 455115:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 8753.953783] Lustre: server umount lustre-MDT0000 complete [ 8755.233269] LustreError: 453444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763518780 with bad export cookie 11199617416775677615 [ 8755.237497] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8755.238456] LustreError: 453444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8755.319623] LustreError: 455317:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 8755.321526] LustreError: 455317:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8755.372226] Lustre: server umount lustre-MDT0001 complete [ 8765.811805] LustreError: 455517:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8765.813672] LustreError: 455517:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8765.862308] Lustre: server umount lustre-OST0000 complete [ 8769.440343] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 21:19:54 (1763518794) [ 8769.809238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8772.174390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8772.355749] LustreError: 456346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8772.363076] LustreError: 456346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8772.381446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8773.774736] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8776.740015] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8776.867889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8778.279732] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8780.189509] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8781.519506] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8783.850502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8783.876306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8783.951293] Lustre: Found index 0 for lustre-OST0000, updating log [ 8785.926135] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8787.731565] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8797.350035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8802.788402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8802.793159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8802.799566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8802.802967] Lustre: Skipped 1 previous similar message [ 8805.740102] LustreError: 458626:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8805.742035] LustreError: 458626:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8805.785021] Lustre: server umount lustre-OST0000 complete [ 8806.744779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8807.904228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8807.908533] LustreError: Skipped 1 previous similar message [ 8807.910895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8807.917822] Lustre: Skipped 1 previous similar message [ 8807.920578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8807.922682] Lustre: Skipped 2 previous similar messages [ 8813.501625] LustreError: 458880:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8813.503629] LustreError: 458880:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8813.532747] Lustre: server umount lustre-MDT0000 complete [ 8814.857308] LustreError: 456327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763518840 with bad export cookie 11199617416775679008 [ 8814.860588] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8814.863554] LustreError: 456327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8815.009593] Lustre: server umount lustre-MDT0001 complete [ 8817.944418] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 21:20:43 (1763518843) [ 8818.302715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8818.654088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8819.005951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8819.369207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8819.742656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8820.095545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8820.462836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8820.840978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8821.216066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8821.577711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8821.938287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8822.266399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8822.595673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8822.951441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8823.299326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8823.632818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8823.981498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8824.311452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8824.640987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8824.977799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8825.338072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8825.676489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8826.000305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8826.322366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8826.662196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8826.993119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8827.315049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8827.654756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8827.988748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8828.318061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8828.644437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8828.986138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8829.316795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8829.655468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8830.011645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8830.358403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8830.702130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8831.035056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8831.372161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8831.709261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8832.046083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8832.368673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8832.678937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8832.993279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8833.315799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8833.652793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8834.001724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8834.327624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8834.650495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8834.992455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8835.306638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8837.864427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8838.008698] LustreError: 462463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8838.031167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8838.033434] Lustre: Skipped 1 previous similar message [ 8839.323198] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8842.203514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8843.746713] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8844.959295] LNet: 1 local NIs in recovery (showing 1): 192.168.204.115@tcp [ 8845.575877] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8846.744651] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8848.896639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8848.929664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8849.029206] Lustre: Found index 0 for lustre-OST0000, updating log [ 8850.150105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 8851.017200] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8852.785796] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8855.519806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8855.521933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8855.527025] Lustre: Skipped 1 previous similar message [ 8855.529523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8855.531265] Lustre: Skipped 4 previous similar messages [ 8859.566057] LustreError: 464779:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8859.568612] LustreError: 464779:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8859.604334] Lustre: server umount lustre-OST0000 complete [ 8861.856856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8861.860819] Lustre: Skipped 2 previous similar messages [ 8863.904713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8863.906339] Lustre: Skipped 6 previous similar messages [ 8866.961208] Lustre: server umount lustre-MDT0000 complete [ 8866.976049] LustreError: 462481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8866.980436] LustreError: 462481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8868.152209] LustreError: 462444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763518893 with bad export cookie 11199617416775680079 [ 8868.155202] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8868.155485] LustreError: 462444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8868.266734] Lustre: server umount lustre-MDT0001 complete [ 8869.155035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8874.121039] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 8877.478422] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8880.835912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8882.990536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8885.068744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8887.342911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8889.800504] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 21:21:55 (1763518915) [ 8905.375293] LNet: 1 local NIs in recovery (showing 1): 192.168.204.115@tcp [ 8920.475752] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8921.456129] Key type lgssc unregistered [ 8921.583486] LNet: 468763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8921.586156] LNetError: 468763:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9049.574627] LNet: Removed LNI 192.168.204.115@tcp [ 9049.884325] Key type .llcrypt unregistered [ 9049.885290] Key type ._llcrypt unregistered [ 9056.755791] Key type ._llcrypt registered [ 9056.756785] Key type .llcrypt registered [ 9056.802781] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9057.206943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9057.259273] alg: No test for adler32 (adler32-zlib) [ 9058.117800] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 9058.206403] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 9059.791162] Key type lgssc registered [ 9060.191745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9061.714718] LNet: Added LNI 192.168.204.115@tcp100 [8/256/0/180] [ 9062.032379] LNet: Added LNI 192.168.204.115@tcp101 [8/256/0/180] [ 9062.333327] LNet: Added LNI 192.168.204.115@tcp102 [8/256/0/180] [ 9062.649141] LNet: Added LNI 192.168.204.115@tcp103 [8/256/0/180] [ 9062.954037] LNet: Added LNI 192.168.204.115@tcp104 [8/256/0/180] [ 9063.256986] LNet: Added LNI 192.168.204.115@tcp105 [8/256/0/180] [ 9063.569245] LNet: Added LNI 192.168.204.115@tcp106 [8/256/0/180] [ 9063.876565] LNet: Added LNI 192.168.204.115@tcp107 [8/256/0/180] [ 9064.174140] LNet: Added LNI 192.168.204.115@tcp108 [8/256/0/180] [ 9064.478532] LNet: Added LNI 192.168.204.115@tcp109 [8/256/0/180] [ 9064.792192] LNet: Added LNI 192.168.204.115@tcp110 [8/256/0/180] [ 9065.097375] LNet: Added LNI 192.168.204.115@tcp111 [8/256/0/180] [ 9065.394736] LNet: Added LNI 192.168.204.115@tcp112 [8/256/0/180] [ 9065.690086] LNet: Added LNI 192.168.204.115@tcp113 [8/256/0/180] [ 9066.003362] LNet: Added LNI 192.168.204.115@tcp114 [8/256/0/180] [ 9066.295087] LNet: Added LNI 192.168.204.115@tcp115 [8/256/0/180] [ 9066.592821] LNet: Added LNI 192.168.204.115@tcp116 [8/256/0/180] [ 9066.889660] LNet: Added LNI 192.168.204.115@tcp117 [8/256/0/180] [ 9067.191846] LNet: Added LNI 192.168.204.115@tcp118 [8/256/0/180] [ 9067.483850] LNet: Added LNI 192.168.204.115@tcp119 [8/256/0/180] [ 9067.774743] LNet: Added LNI 192.168.204.115@tcp120 [8/256/0/180] [ 9068.069829] LNet: Added LNI 192.168.204.115@tcp121 [8/256/0/180] [ 9068.368148] LNet: Added LNI 192.168.204.115@tcp122 [8/256/0/180] [ 9068.663671] LNet: Added LNI 192.168.204.115@tcp123 [8/256/0/180] [ 9068.965186] LNet: Added LNI 192.168.204.115@tcp124 [8/256/0/180] [ 9069.281364] LNet: Added LNI 192.168.204.115@tcp125 [8/256/0/180] [ 9069.597622] LNet: Added LNI 192.168.204.115@tcp126 [8/256/0/180] [ 9069.886566] LNet: Added LNI 192.168.204.115@tcp127 [8/256/0/180] [ 9070.183906] LNet: Added LNI 192.168.204.115@tcp128 [8/256/0/180] [ 9070.480776] LNet: Added LNI 192.168.204.115@tcp129 [8/256/0/180] [ 9070.763377] LNet: Added LNI 192.168.204.115@tcp130 [8/256/0/180] [ 9071.057900] LNet: Added LNI 192.168.204.115@tcp131 [8/256/0/180] [ 9071.367460] LNet: Added LNI 192.168.204.115@tcp132 [8/256/0/180] [ 9071.665179] LNet: Added LNI 192.168.204.115@tcp133 [8/256/0/180] [ 9071.969475] LNet: Added LNI 192.168.204.115@tcp134 [8/256/0/180] [ 9072.255160] LNet: Added LNI 192.168.204.115@tcp135 [8/256/0/180] [ 9074.488178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9074.509414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9074.514739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9075.555183] Lustre: 471752:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.115@tcp: can't add NID 192.168.204.115@tcp130: rc = -75 [ 9075.584297] LNetError: 471774:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [ 9075.587442] LNetError: Protocol error connecting to 192.168.204.115@tcp116 at host 192.168.204.115:988: is it running a compatible version of Lustre? [ 9075.639947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9075.653444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9075.697090] Lustre: lustre-MDT0000: new disk, initializing [ 9075.731795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9075.741113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9077.133350] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9081.405311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9081.430081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9081.473178] Lustre: 471771:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9081.502926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9081.505218] Lustre: Skipped 1 previous similar message [ 9081.548897] Lustre: lustre-MDT0001: new disk, initializing [ 9081.564119] Lustre: 472556:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.115@tcp: can't add NID 192.168.204.115@tcp130: rc = -75 [ 9081.567429] Lustre: 472556:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9081.619549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9081.636288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9081.639755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9083.008505] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9086.168122] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9087.373303] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9089.632421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9089.658444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9089.777871] Lustre: lustre-OST0000: new disk, initializing [ 9089.780320] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9089.812540] Lustre: 473523:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.115@tcp: can't add NID 192.168.204.115@tcp130: rc = -75 [ 9089.815214] Lustre: 473523:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9089.830780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9090.234073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9090.239099] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9090.315176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9091.803554] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9094.966726] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9100.770923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9100.770993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9100.771689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9100.773127] LustreError: Skipped 1 previous similar message [ 9100.778061] Lustre: Skipped 1 previous similar message [ 9103.533101] LustreError: 474359:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 9103.569865] Lustre: server umount lustre-OST0000 complete [ 9105.889656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9105.890201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9105.894788] Lustre: Skipped 1 previous similar message [ 9105.897382] Lustre: Skipped 1 previous similar message [ 9110.432431] LNetError: 469213:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [ 9110.437570] LNetError: Protocol error connecting to 192.168.204.115@tcp116 at host 192.168.204.115:988: is it running a compatible version of Lustre? [ 9111.008329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9111.011710] Lustre: Skipped 1 previous similar message [ 9116.128151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9116.130226] Lustre: Skipped 1 previous similar message [ 9119.199125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9119.238463] LustreError: 474560:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9119.241877] LustreError: 474560:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9119.272467] Lustre: server umount lustre-MDT0000 complete [ 9120.530835] LustreError: 472555:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763519145 with bad export cookie 7266133647178821440 [ 9120.536501] LustreError: 472555:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9120.536823] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9120.610714] LustreError: 474762:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 9120.612713] LustreError: 474762:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9120.673962] Lustre: server umount lustre-MDT0001 complete [ 9122.720060] LNetError: 469216:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [ 9122.724168] LNetError: Protocol error connecting to 192.168.204.115@tcp116 at host 192.168.204.115:988: is it running a compatible version of Lustre? [ 9124.557823] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 9125.593646] Key type lgssc unregistered [ 9125.772827] LNet: 475301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9125.777167] LNetError: 475301:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9125.781048] LNet: Removed LNI 192.168.204.115@tcp [ 9126.815167] LNet: Removed LNI 192.168.204.115@tcp100 [ 9126.817356] LNetError: 475301:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9126.820952] LNetError: 475301:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9126.824867] LNet: Removed LNI 192.168.204.115@tcp101 [ 9127.839128] LNet: Removed LNI 192.168.204.115@tcp102 [ 9127.840489] LNetError: 475301:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9127.843954] LNetError: 475301:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9127.846101] LNet: Removed LNI 192.168.204.115@tcp103 [ 9128.863149] LNet: Removed LNI 192.168.204.115@tcp104 [ 9129.887160] LNet: Removed LNI 192.168.204.115@tcp105 [ 9129.889151] LNetError: 475301:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9129.892293] LNetError: 475301:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 9129.895635] LNet: Removed LNI 192.168.204.115@tcp106 [ 9130.911158] LNet: Removed LNI 192.168.204.115@tcp107 [ 9131.935163] LNet: Removed LNI 192.168.204.115@tcp108 [ 9132.959195] LNet: Removed LNI 192.168.204.115@tcp109 [ 9133.983120] LNet: Removed LNI 192.168.204.115@tcp110 [ 9133.984506] LNetError: 475301:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9133.986683] LNetError: 475301:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 9133.989855] LNet: Removed LNI 192.168.204.115@tcp111 [ 9135.007128] LNet: Removed LNI 192.168.204.115@tcp112 [ 9136.031121] LNet: Removed LNI 192.168.204.115@tcp113 [ 9137.055146] LNet: Removed LNI 192.168.204.115@tcp114 [ 9138.079167] LNet: Removed LNI 192.168.204.115@tcp115 [ 9139.103159] LNet: Removed LNI 192.168.204.115@tcp116 [ 9139.104886] LNet: Removed LNI 192.168.204.115@tcp117 [ 9139.106170] LNet: Removed LNI 192.168.204.115@tcp118 [ 9139.107520] LNet: Removed LNI 192.168.204.115@tcp119 [ 9139.108797] LNet: Removed LNI 192.168.204.115@tcp120 [ 9139.110063] LNet: Removed LNI 192.168.204.115@tcp121 [ 9139.111316] LNet: Removed LNI 192.168.204.115@tcp122 [ 9139.112616] LNet: Removed LNI 192.168.204.115@tcp123 [ 9139.113904] LNet: Removed LNI 192.168.204.115@tcp124 [ 9139.115068] LNet: Removed LNI 192.168.204.115@tcp125 [ 9139.116239] LNet: Removed LNI 192.168.204.115@tcp126 [ 9139.117431] LNet: Removed LNI 192.168.204.115@tcp127 [ 9139.118626] LNet: Removed LNI 192.168.204.115@tcp128 [ 9139.119856] LNet: Removed LNI 192.168.204.115@tcp129 [ 9139.121073] LNet: Removed LNI 192.168.204.115@tcp130 [ 9139.122366] LNet: Removed LNI 192.168.204.115@tcp131 [ 9139.123548] LNet: Removed LNI 192.168.204.115@tcp132 [ 9139.124917] LNet: Removed LNI 192.168.204.115@tcp133 [ 9139.126177] LNet: Removed LNI 192.168.204.115@tcp134 [ 9139.133534] LNet: Removed LNI 192.168.204.115@tcp135 [ 9139.403804] Key type .llcrypt unregistered [ 9139.405524] Key type ._llcrypt unregistered [ 9147.029863] Key type ._llcrypt registered [ 9147.031047] Key type .llcrypt registered [ 9147.070214] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 9153.582862] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9153.985578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9154.041226] alg: No test for adler32 (adler32-zlib) [ 9154.888542] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 9154.964972] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 9156.543148] Key type lgssc registered [ 9156.913707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9159.914910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9162.131641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9164.204477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9166.342579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9168.727834] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 21:26:34 (1763519194) [ 9174.942865] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 9176.059702] Key type lgssc unregistered [ 9176.194458] LNet: 479312:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9176.197609] LNetError: 479312:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9176.208293] LNet: Removed LNI 192.168.204.115@tcp [ 9176.525765] Key type .llcrypt unregistered [ 9176.526836] Key type ._llcrypt unregistered [ 9183.415439] Key type ._llcrypt registered [ 9183.416568] Key type .llcrypt registered [ 9183.452276] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9183.743065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9183.753256] alg: No test for adler32 (adler32-zlib) [ 9184.634163] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 9184.729523] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 9186.319145] Key type lgssc registered [ 9186.665281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9189.648164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9189.669976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9189.674929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9190.760981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9190.773095] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9190.809080] Lustre: lustre-MDT0000: new disk, initializing [ 9190.836078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9190.842944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9192.197831] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9196.400379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9196.421781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9196.456176] Lustre: 480484:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9196.505509] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9196.508420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9196.508426] Lustre: Skipped 1 previous similar message [ 9196.543540] Lustre: lustre-MDT0001: new disk, initializing [ 9196.580916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9196.597198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9196.600729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9197.896056] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9201.014449] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9202.294656] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9204.581803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9204.607272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9204.717343] Lustre: lustre-OST0000: new disk, initializing [ 9204.719281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9204.746670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9206.548849] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9208.573785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9208.577393] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9208.617405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9209.685627] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9210.844488] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing lnet_if_list [ 9212.445956] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 9212.458175] LNet: Added LNI 192.168.204.115@tcp10 [8/256/0/180] [ 9214.269601] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 9214.280933] LNet: Added LNI 192.168.204.115@tcp11 [8/256/0/180] [ 9215.727930] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 9215.743717] LNet: Added LNI 192.168.204.115@tcp12 [8/256/0/180] [ 9217.285214] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 9217.297260] LNet: Added LNI 192.168.204.115@tcp13 [8/256/0/180] [ 9218.622618] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 9218.631597] LNet: Added LNI 192.168.204.115@tcp14 [8/256/0/180] [ 9220.039996] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 9220.049867] LNet: Added LNI 192.168.204.115@tcp15 [8/256/0/180] [ 9221.545112] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 9221.557536] LNet: Added LNI 192.168.204.115@tcp16 [8/256/0/180] [ 9223.071986] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 9223.085269] LNet: Added LNI 192.168.204.115@tcp17 [8/256/0/180] [ 9224.655205] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 9224.669951] LNet: Added LNI 192.168.204.115@tcp18 [8/256/0/180] [ 9226.138875] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 9226.149243] LNet: Added LNI 192.168.204.115@tcp19 [8/256/0/180] [ 9227.662981] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 9227.676799] LNet: Added LNI 192.168.204.115@tcp20 [8/256/0/180] [ 9229.224322] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 9229.234794] LNet: Added LNI 192.168.204.115@tcp21 [8/256/0/180] [ 9230.859866] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 9230.870155] LNet: Added LNI 192.168.204.115@tcp22 [8/256/0/180] [ 9232.516464] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 9232.526347] LNet: Added LNI 192.168.204.115@tcp23 [8/256/0/180] [ 9234.196802] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 9234.208846] LNet: Added LNI 192.168.204.115@tcp24 [8/256/0/180] [ 9235.943475] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 9235.953188] LNet: Added LNI 192.168.204.115@tcp25 [8/256/0/180] [ 9237.839336] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 9237.849658] LNet: Added LNI 192.168.204.115@tcp26 [8/256/0/180] [ 9239.617194] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 9239.626954] LNet: Added LNI 192.168.204.115@tcp27 [8/256/0/180] [ 9239.634395] LNetError: 479745:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [ 9239.638611] LNetError: Protocol error connecting to 192.168.204.115@tcp26 at host 192.168.204.115:988: is it running a compatible version of Lustre? [ 9241.615471] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 9241.625645] LNet: Added LNI 192.168.204.115@tcp28 [8/256/0/180] [ 9241.630169] LNetError: 479744:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [ 9241.636502] LNetError: Protocol error connecting to 192.168.204.115@tcp26 at host 192.168.204.115:988: is it running a compatible version of Lustre? [ 9241.639237] LNetError: Skipped 1 previous similar message [ 9243.633972] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 9243.644972] LNet: Added LNI 192.168.204.115@tcp29 [8/256/0/180] [ 9243.648970] LNetError: 479745:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [ 9243.651833] LNetError: 479745:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [ 9243.654453] LNetError: Protocol error connecting to 192.168.204.115@tcp27 at host 192.168.204.115:988: is it running a compatible version of Lustre? [ 9245.722135] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 9245.732837] LNet: Added LNI 192.168.204.115@tcp30 [8/256/0/180] [ 9245.739805] LNetError: 479742:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [ 9245.743818] LNetError: Protocol error connecting to 192.168.204.115@tcp29 at host 192.168.204.115:988: is it running a compatible version of Lustre? [ 9246.176266] LNet: 1 local NIs in recovery (showing 1): 192.168.204.115@tcp29 [ 9247.786084] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 9247.796358] LNet: Added LNI 192.168.204.115@tcp31 [8/256/0/180] [ 9249.938330] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 9249.951265] LNet: Added LNI 192.168.204.115@tcp32 [8/256/0/180] [ 9249.956485] LNetError: 479744:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [ 9249.960162] LNetError: 479744:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 4 previous similar messages [ 9249.962907] LNetError: Protocol error connecting to 192.168.204.115@tcp31 at host 192.168.204.115:988: is it running a compatible version of Lustre? [ 9249.966253] LNetError: Skipped 4 previous similar messages [ 9252.163930] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 9252.177386] LNet: Added LNI 192.168.204.115@tcp33 [8/256/0/180] [ 9254.411745] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 9254.423847] LNet: Added LNI 192.168.204.115@tcp34 [8/256/0/180] [ 9256.723612] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 9256.738022] LNet: Added LNI 192.168.204.115@tcp35 [8/256/0/180] [ 9258.926756] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 9258.937363] LNet: Added LNI 192.168.204.115@tcp36 [8/256/0/180] [ 9259.488645] LNetError: 479744:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [ 9259.491763] LNetError: 479744:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 12 previous similar messages [ 9259.494255] LNetError: Protocol error connecting to 192.168.204.115@tcp34 at host 192.168.204.115:988: is it running a compatible version of Lustre? [ 9259.497588] LNetError: Skipped 12 previous similar messages [ 9261.103987] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 9261.114820] LNet: Added LNI 192.168.204.115@tcp37 [8/256/0/180] [ 9263.517915] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 9263.532601] LNet: Added LNI 192.168.204.115@tcp38 [8/256/0/180] [ 9265.876751] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 9265.888853] LNet: Added LNI 192.168.204.115@tcp39 [8/256/0/180] [ 9268.159492] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 9268.173887] LNet: Added LNI 192.168.204.115@tcp40 [8/256/0/180] [ 9270.505639] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 9270.517063] LNet: Added LNI 192.168.204.115@tcp41 [8/256/0/180] [ 9272.989977] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 9273.001749] LNet: Added LNI 192.168.204.115@tcp42 [8/256/0/180] [ 9275.666448] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 9275.680766] LNet: Added LNI 192.168.204.115@tcp43 [8/256/0/180] [ 9276.899784] LNetError: 479744:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [ 9276.903273] LNetError: 479744:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 21 previous similar messages [ 9276.906369] LNetError: Protocol error connecting to 192.168.204.115@tcp39 at host 192.168.204.115:988: is it running a compatible version of Lustre? [ 9276.910337] LNetError: Skipped 23 previous similar messages [ 9278.387191] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 9278.398145] LNet: Added LNI 192.168.204.115@tcp44 [8/256/0/180] [ 9281.125601] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 9281.138589] LNet: Added LNI 192.168.204.115@tcp45 [8/256/0/180] [ 9283.797119] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 9283.809606] LNet: Added LNI 192.168.204.115@tcp46 [8/256/0/180] [ 9286.486052] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 9286.501179] LNet: Added LNI 192.168.204.115@tcp47 [8/256/0/180] [ 9289.076593] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 9289.087030] LNet: Added LNI 192.168.204.115@tcp48 [8/256/0/180] [ 9291.848634] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 9291.857864] LNet: Added LNI 192.168.204.115@tcp49 [8/256/0/180] [ 9294.515533] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 9294.524840] LNet: Added LNI 192.168.204.115@tcp50 [8/256/0/180] [ 9304.038044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9304.042550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9304.044398] Lustre: Skipped 1 previous similar message [ 9306.591408] LNet: 6 local NIs in recovery (showing 6): 192.168.204.115@tcp46, 192.168.204.115@tcp47, 192.168.204.115@tcp42, 192.168.204.115@tcp40, 192.168.204.115@tcp34, 192.168.204.115@tcp30 [ 9307.565707] LustreError: 504667:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9307.601806] Lustre: server umount lustre-OST0000 complete [ 9309.154327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9309.155334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9309.157746] Lustre: Skipped 2 previous similar messages [ 9309.160816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9310.689314] LNetError: 499311:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [ 9310.691889] LNetError: 499311:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 54 previous similar messages [ 9310.693825] LNetError: Protocol error connecting to 192.168.204.115@tcp42 at host 192.168.204.115:988: is it running a compatible version of Lustre? [ 9310.696310] LNetError: Skipped 51 previous similar messages [ 9314.272203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9314.273879] Lustre: Skipped 2 previous similar messages [ 9314.940418] LustreError: 504869:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9314.942227] LustreError: 504869:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9314.973353] Lustre: server umount lustre-MDT0000 complete [ 9316.182989] LustreError: 480475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763519341 with bad export cookie 13038487528985570904 [ 9316.186188] LustreError: 480475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9316.186196] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9316.248819] LustreError: 505070:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9316.250646] LustreError: 505070:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9316.300579] Lustre: server umount lustre-MDT0001 complete [ 9319.773457] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 9320.810234] Key type lgssc unregistered [ 9320.958865] LNet: 505609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9320.962421] LNetError: 505609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9320.964865] LNet: Removed LNI 192.168.204.115@tcp [ 9322.015168] LNet: Removed LNI 192.168.204.115@tcp10 [ 9322.017306] LNetError: 505609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9322.020908] LNetError: 505609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9322.024878] LNet: Removed LNI 192.168.204.115@tcp11 [ 9323.039112] LNet: Removed LNI 192.168.204.115@tcp12 [ 9323.040380] LNetError: 505609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9323.042605] LNetError: 505609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9323.044646] LNet: Removed LNI 192.168.204.115@tcp13 [ 9324.063119] LNet: Removed LNI 192.168.204.115@tcp14 [ 9325.087100] LNet: Removed LNI 192.168.204.115@tcp15 [ 9325.088517] LNetError: 505609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9325.090513] LNetError: 505609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 9325.092829] LNet: Removed LNI 192.168.204.115@tcp16 [ 9326.111162] LNet: Removed LNI 192.168.204.115@tcp17 [ 9327.135153] LNet: Removed LNI 192.168.204.115@tcp18 [ 9328.159207] LNet: Removed LNI 192.168.204.115@tcp19 [ 9329.183111] LNet: Removed LNI 192.168.204.115@tcp20 [ 9329.184428] LNetError: 505609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9329.186342] LNetError: 505609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 9329.188448] LNet: Removed LNI 192.168.204.115@tcp21 [ 9330.207183] LNet: Removed LNI 192.168.204.115@tcp22 [ 9331.231152] LNet: Removed LNI 192.168.204.115@tcp23 [ 9332.255151] LNet: Removed LNI 192.168.204.115@tcp24 [ 9333.279112] LNet: Removed LNI 192.168.204.115@tcp25 [ 9334.303178] LNet: Removed LNI 192.168.204.115@tcp26 [ 9335.327123] LNet: Removed LNI 192.168.204.115@tcp27 [ 9336.351371] LNet: Removed LNI 192.168.204.115@tcp28 [ 9337.375111] LNet: Removed LNI 192.168.204.115@tcp29 [ 9337.376339] LNetError: 505609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9337.378255] LNetError: 505609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 9337.380374] LNet: Removed LNI 192.168.204.115@tcp30 [ 9338.399141] LNet: Removed LNI 192.168.204.115@tcp31 [ 9339.423168] LNet: Removed LNI 192.168.204.115@tcp32 [ 9340.447157] LNet: Removed LNI 192.168.204.115@tcp33 [ 9340.448368] LNet: Removed LNI 192.168.204.115@tcp34 [ 9341.471120] LNet: Removed LNI 192.168.204.115@tcp35 [ 9342.495128] LNet: Removed LNI 192.168.204.115@tcp36 [ 9343.519136] LNet: Removed LNI 192.168.204.115@tcp37 [ 9344.543123] LNet: Removed LNI 192.168.204.115@tcp38 [ 9345.567171] LNet: Removed LNI 192.168.204.115@tcp39 [ 9346.591116] LNet: Removed LNI 192.168.204.115@tcp40 [ 9347.615110] LNet: Removed LNI 192.168.204.115@tcp41 [ 9347.616337] LNet: Removed LNI 192.168.204.115@tcp42 [ 9348.639100] LNet: Removed LNI 192.168.204.115@tcp43 [ 9349.663118] LNet: Removed LNI 192.168.204.115@tcp44 [ 9350.687102] LNet: Removed LNI 192.168.204.115@tcp45 [ 9350.688379] LNet: Removed LNI 192.168.204.115@tcp46 [ 9350.689599] LNet: Removed LNI 192.168.204.115@tcp47 [ 9350.690719] LNet: Removed LNI 192.168.204.115@tcp48 [ 9350.691877] LNet: Removed LNI 192.168.204.115@tcp49 [ 9350.699408] LNet: Removed LNI 192.168.204.115@tcp50 [ 9350.930382] Key type .llcrypt unregistered [ 9350.932174] Key type ._llcrypt unregistered [ 9358.177918] Key type ._llcrypt registered [ 9358.178987] Key type .llcrypt registered [ 9358.214896] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9358.564496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9358.582135] alg: No test for adler32 (adler32-zlib) [ 9359.434835] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 9359.522143] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 9361.103156] Key type lgssc registered [ 9361.454115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9363.920141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9363.926814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.090151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9366.429609] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9369.243239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9369.374136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9370.611891] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9372.307640] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9373.554991] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9375.693403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9375.807699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9377.705652] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9379.437827] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9398.408179] loop0: detected capacity change from 0 to 204800000 [ 9398.523307] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9398.525874] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9398.529280] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9398.548224] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9400.480710] loop0: detected capacity change from 0 to 204800000 [ 9400.533666] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9400.537146] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9400.541521] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9400.587054] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9402.162234] loop0: detected capacity change from 0 to 204800000 [ 9404.111920] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9404.134508] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9404.181693] Lustre: fs15246-OST0000: new disk, initializing [ 9404.184933] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 9404.203332] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9406.093054] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9409.916554] loop1: detected capacity change from 0 to 204800000 [ 9411.784410] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9411.810157] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9411.838921] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 9411.865062] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 9411.904619] Lustre: fs15246-MDT0000: new disk, initializing [ 9411.958016] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9411.967860] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9412.021499] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9413.088054] Lustre: 511230:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000070d51e2d x1849184101629184/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763519454 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 9413.322518] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9422.175195] LustreError: 511230:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 9422.178203] LustreError: 511230:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 9422.182198] LustreError: 511230:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 9424.355751] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9424.361670] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 9426.876629] LustreError: 512353:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 9426.900942] Lustre: server umount fs15246-MDT0000 complete [ 9429.478414] LustreError: 506828:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9429.482492] LustreError: 506828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9429.807119] LustreError: 512855:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 9429.809725] LustreError: 512855:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9429.825962] Lustre: server umount fs15246-OST0000 complete [ 9434.593936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9434.594958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9434.596903] Lustre: Skipped 1 previous similar message [ 9434.599405] Lustre: Skipped 1 previous similar message [ 9439.712856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9439.716060] Lustre: Skipped 1 previous similar message [ 9444.837487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9444.839831] Lustre: Skipped 1 previous similar message [ 9447.391168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9447.409218] LustreError: 513356:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9447.410859] LustreError: 513356:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 9447.448989] Lustre: server umount lustre-OST0000 complete [ 9449.953082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9449.954321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9449.956239] Lustre: Skipped 1 previous similar message [ 9454.786136] LustreError: 513559:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9454.787851] LustreError: 513559:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9454.814394] Lustre: server umount lustre-MDT0000 complete [ 9455.072025] LustreError: 507558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9455.075919] LustreError: 507558:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9456.003774] LustreError: 506813:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763519481 with bad export cookie 14125739954174779902 [ 9456.007783] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9456.009337] LustreError: 506813:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9456.137426] Lustre: server umount lustre-MDT0001 complete [ 9459.655205] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 9460.733383] Key type lgssc unregistered [ 9460.855418] LNet: 514293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9460.857585] LNetError: 514293:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9460.867360] LNet: Removed LNI 192.168.204.115@tcp [ 9461.185786] Key type .llcrypt unregistered [ 9461.187027] Key type ._llcrypt unregistered [ 9468.481792] Key type ._llcrypt registered [ 9468.483169] Key type .llcrypt registered [ 9468.526957] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9468.902419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9468.914205] alg: No test for adler32 (adler32-zlib) [ 9469.780855] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 9469.876068] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 9471.463185] Key type lgssc registered [ 9471.836176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9474.357822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9475.943175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9477.587234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9479.166823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9484.182520] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 9487.181557] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9490.522254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9492.567297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9494.563737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9496.713747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9498.961814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9498.985666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9498.989765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9500.086539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9500.099849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9500.138254] Lustre: lustre-MDT0000: new disk, initializing [ 9500.164215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9500.170191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9501.378807] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9505.519743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9505.544274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9505.582934] Lustre: 518695:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9505.613764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9505.616870] Lustre: Skipped 1 previous similar message [ 9505.661463] Lustre: lustre-MDT0001: new disk, initializing [ 9505.700709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9505.712372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9505.716231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9506.949457] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9509.868347] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9511.106516] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9513.383389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9513.407889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9513.522622] Lustre: lustre-OST0000: new disk, initializing [ 9513.526256] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9513.560213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9515.347624] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9518.077588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9518.081276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9518.118223] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9518.412901] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9519.720301] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9519.790245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9521.010357] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9521.083981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9523.172866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9523.173197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9523.174167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9523.175301] LustreError: Skipped 1 previous similar message [ 9523.186361] Lustre: Skipped 1 previous similar message [ 9527.791109] LustreError: 521632:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9527.832254] Lustre: server umount lustre-OST0000 complete [ 9531.359937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9531.363485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9531.369609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9531.371119] Lustre: Skipped 1 previous similar message [ 9533.408118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9533.408584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9535.168460] LustreError: 521833:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9535.172223] LustreError: 521833:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9535.204912] Lustre: server umount lustre-MDT0000 complete [ 9536.402475] LustreError: 518687:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763519561 with bad export cookie 12254320853655032714 [ 9536.406072] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9536.407791] LustreError: 518687:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9536.480580] LustreError: 522034:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9536.482356] LustreError: 522034:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9536.531884] Lustre: server umount lustre-MDT0001 complete [ 9544.058186] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 9547.288796] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9550.626951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9552.899068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9554.915719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9556.971896] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9557.494347] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9353 sec ========== 21:33:02 (1763519582) [ 9558.020527] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:33:03 (1763519583) === [ 9559.143503] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:33:04 (1763519584) === [ 9571.920768] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 9572.965485] Key type lgssc unregistered [ 9573.085365] LNet: 526880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9573.089430] LNetError: 526880:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9573.102277] LNet: Removed LNI 192.168.204.115@tcp [ 9573.385260] Key type .llcrypt unregistered [ 9573.387090] Key type ._llcrypt unregistered