[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 465397520 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002264] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.006783] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007013] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008005] pid_max: default: 32768 minimum: 301 [ 0.009115] LSM: Security Framework initializing [ 0.010027] Yama: becoming mindful. [ 0.010708] SELinux: Initializing. [ 0.011043] *** VALIDATE selinux *** [ 0.018826] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023467] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024111] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026041] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027073] *** VALIDATE tmpfs *** [ 0.028429] *** VALIDATE proc *** [ 0.029192] *** VALIDATE cgroup *** [ 0.030004] *** VALIDATE cgroup2 *** [ 0.031210] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032139] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034022] Spectre V2 : User space: Vulnerable [ 0.035005] Speculative Store Bypass: Vulnerable [ 0.037836] debug: unmapping init [mem 0xffffffff97c59000-0xffffffff97c60fff] [ 0.039938] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040536] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041014] ... version: 2 [ 0.042005] ... bit width: 48 [ 0.043007] ... generic registers: 4 [ 0.044007] ... value mask: 0000ffffffffffff [ 0.045005] ... max period: 00007fffffffffff [ 0.046006] ... fixed-purpose events: 3 [ 0.047006] ... event mask: 000000070000000f [ 0.048229] rcu: Hierarchical SRCU implementation. [ 0.050313] smp: Bringing up secondary CPUs ... [ 0.051502] x86: Booting SMP configuration: [ 0.052010] .... node #0, CPUs: #1 #2 #3 [ 0.055203] smp: Brought up 1 node, 4 CPUs [ 0.057006] smpboot: Max logical packages: 1 [ 0.057795] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.202593] node 0 deferred pages initialised in 144ms [ 0.207015] devtmpfs: initialized [ 0.208191] x86/mm: Memory block size: 128MB [ 0.210668] gcov: version magic: 0x41383552 [ 0.213180] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.215056] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.217239] pinctrl core: initialized pinctrl subsystem [ 0.219110] [ 0.219589] ************************************************************* [ 0.221006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.223007] ** ** [ 0.225006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.227008] ** ** [ 0.228005] ** This means that this kernel is built to expose internal ** [ 0.230006] ** IOMMU data structures, which may compromise security on ** [ 0.231005] ** your system. ** [ 0.233006] ** ** [ 0.235008] ** If you see this message and you are not debugging the ** [ 0.236006] ** kernel, report this immediately to your vendor! ** [ 0.238006] ** ** [ 0.239005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.241006] ************************************************************* [ 0.242755] NET: Registered protocol family 16 [ 0.244356] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.247029] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.249032] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.252524] cpuidle: using governor menu [ 0.254473] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.256425] PCI: Using configuration type 1 for base access [ 0.258128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.266084] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.267015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.270085] cryptd: max_cpu_qlen set to 1000 [ 0.272199] ACPI: Added _OSI(Module Device) [ 0.274010] ACPI: Added _OSI(Processor Device) [ 0.274898] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.276010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.280190] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.286472] ACPI: Interpreter enabled [ 0.288049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.290010] ACPI: Using IOAPIC for interrupt routing [ 0.291072] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.295361] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.306254] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.309023] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.310008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.312058] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.315510] acpiphp: Slot [2] registered [ 0.317102] acpiphp: Slot [3] registered [ 0.317982] acpiphp: Slot [4] registered [ 0.318042] acpiphp: Slot [5] registered [ 0.318936] acpiphp: Slot [6] registered [ 0.320075] acpiphp: Slot [7] registered [ 0.320984] acpiphp: Slot [8] registered [ 0.322092] acpiphp: Slot [9] registered [ 0.323076] acpiphp: Slot [10] registered [ 0.324083] acpiphp: Slot [11] registered [ 0.325045] acpiphp: Slot [12] registered [ 0.326046] acpiphp: Slot [13] registered [ 0.327046] acpiphp: Slot [14] registered [ 0.328062] acpiphp: Slot [15] registered [ 0.328995] acpiphp: Slot [16] registered [ 0.330045] acpiphp: Slot [17] registered [ 0.331045] acpiphp: Slot [18] registered [ 0.332048] acpiphp: Slot [19] registered [ 0.332946] acpiphp: Slot [20] registered [ 0.335047] acpiphp: Slot [21] registered [ 0.336072] acpiphp: Slot [22] registered [ 0.337046] acpiphp: Slot [23] registered [ 0.338047] acpiphp: Slot [24] registered [ 0.339057] acpiphp: Slot [25] registered [ 0.340082] acpiphp: Slot [26] registered [ 0.342069] acpiphp: Slot [27] registered [ 0.343059] acpiphp: Slot [28] registered [ 0.343984] acpiphp: Slot [29] registered [ 0.345052] acpiphp: Slot [30] registered [ 0.346048] acpiphp: Slot [31] registered [ 0.347024] PCI host bridge to bus 0000:00 [ 0.347928] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.349009] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.350007] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.352008] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.354012] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.357012] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.358150] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.362058] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.365344] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.373010] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.376550] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.379011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.381013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.383010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.385540] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.387504] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.389020] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.390397] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.394008] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.407014] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.413011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.418894] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.424010] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.429015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.440010] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.449894] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.458021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.466016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.487016] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.492000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.499026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.506022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.521016] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.531063] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.537016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.543015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.555013] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.565896] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.570009] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.575010] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.585012] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.591875] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.597016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.603015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.614017] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.621220] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.623251] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.625314] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.627224] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.629169] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.633313] iommu: Default domain type: Passthrough [ 0.635433] SCSI subsystem initialized [ 0.637077] ACPI: bus type USB registered [ 0.638062] usbcore: registered new interface driver usbfs [ 0.639032] usbcore: registered new interface driver hub [ 0.640037] usbcore: registered new device driver usb [ 0.642088] pps_core: LinuxPPS API ver. 1 registered [ 0.643006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.646023] PTP clock support registered [ 0.647209] EDAC MC: Ver: 3.0.0 [ 0.648358] PCI: Using ACPI for IRQ routing [ 0.649625] NetLabel: Initializing [ 0.651006] NetLabel: domain hash size = 128 [ 0.651925] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.653044] NetLabel: unlabeled traffic allowed by default [ 0.655090] vgaarb: loaded [ 0.656269] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.657013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.662208] clocksource: Switched to clocksource kvm-clock [ 0.753904] VFS: Disk quotas dquot_6.6.0 [ 0.755070] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.756862] *** VALIDATE ramfs *** [ 0.757827] *** VALIDATE hugetlbfs *** [ 0.758967] pnp: PnP ACPI init [ 0.760727] pnp: PnP ACPI: found 6 devices [ 0.774595] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.777150] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.778691] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.780177] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.782017] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.783822] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.786139] NET: Registered protocol family 2 [ 0.788137] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.791980] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.794602] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.799239] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.802283] TCP: Hash tables configured (established 65536 bind 65536) [ 0.804480] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.806754] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.808812] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.811221] NET: Registered protocol family 1 [ 0.813686] RPC: Registered named UNIX socket transport module. [ 0.815643] RPC: Registered udp transport module. [ 0.817161] RPC: Registered tcp transport module. [ 0.818789] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.820755] NET: Registered protocol family 44 [ 0.821931] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.823422] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.824633] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.826252] PCI: CLS 0 bytes, default 64 [ 0.827784] Unpacking initramfs... [ 2.066856] debug: unmapping init [mem 0xffff99377cc54000-0xffff99377ffbffff] [ 2.069598] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.071114] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.072939] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.579158] Initialise system trusted keyrings [ 2.580983] Key type blacklist registered [ 2.583721] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.591223] zbud: loaded [ 2.593510] *** VALIDATE nfs *** [ 2.594831] *** VALIDATE nfs4 *** [ 2.596438] pstore: using deflate compression [ 2.599094] Platform Keyring initialized [ 2.680366] NET: Registered protocol family 38 [ 2.682323] Key type asymmetric registered [ 2.684013] Asymmetric key parser 'x509' registered [ 2.686085] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.689307] io scheduler mq-deadline registered [ 2.690466] io scheduler kyber registered [ 2.691751] io scheduler bfq registered [ 2.693079] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.695121] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.697194] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.699474] ACPI: Power Button [PWRF] [ 2.774039] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.868790] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.053436] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.135310] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.285865] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.312521] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.343576] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.350658] Non-volatile memory driver v1.3 [ 3.352404] Linux agpgart interface v0.103 [ 3.387900] virtio_blk virtio1: [vda] 133608 512-byte logical blocks (68.4 MB/65.2 MiB) [ 3.391761] vda: detected capacity change from 0 to 68407296 [ 3.407020] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.409849] vdb: detected capacity change from 0 to 1073741824 [ 3.423301] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.425531] vdc: detected capacity change from 0 to 2621440000 [ 3.447580] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.450874] vdd: detected capacity change from 0 to 2621440000 [ 3.466166] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.468538] vde: detected capacity change from 0 to 4294967296 [ 3.480977] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.483136] vdf: detected capacity change from 0 to 4294967296 [ 3.489305] libphy: Fixed MDIO Bus: probed [ 3.494211] usbcore: registered new interface driver usbserial_generic [ 3.497252] usbserial: USB Serial support registered for generic [ 3.500086] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.504842] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.507317] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.510230] mousedev: PS/2 mouse device common for all mice [ 3.513383] rtc_cmos 00:05: RTC can wake from S4 [ 3.515869] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.521961] rtc_cmos 00:05: registered as rtc0 [ 3.523605] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.524436] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.530092] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.530340] intel_pstate: CPU model not supported [ 3.536250] hid: raw HID events driver (C) Jiri Kosina [ 3.538781] usbcore: registered new interface driver usbhid [ 3.540200] usbhid: USB HID core driver [ 3.541288] drop_monitor: Initializing network drop monitor service [ 3.542836] Initializing XFRM netlink socket [ 3.544268] NET: Registered protocol family 10 [ 3.546730] Segment Routing with IPv6 [ 3.548544] NET: Registered protocol family 17 [ 3.551913] mpls_gso: MPLS GSO support [ 3.558329] RAS: Correctable Errors collector initialized. [ 3.559836] AVX version of gcm_enc/dec engaged. [ 3.560970] AES CTR mode by8 optimization enabled [ 3.626375] sched_clock: Marking stable (3626344648, 0)->(4333497358, -707152710) [ 3.629663] registered taskstats version 1 [ 3.631743] Loading compiled-in X.509 certificates [ 3.634196] zswap: loaded using pool lzo/zbud [ 3.658771] Key type big_key registered [ 3.671444] Key type encrypted registered [ 3.673300] ima: No TPM chip found, activating TPM-bypass! [ 3.675423] ima: Allocated hash algorithm: sha1 [ 3.677130] ima: No architecture policies found [ 3.678780] evm: Initialising EVM extended attributes: [ 3.680681] evm: security.selinux [ 3.681835] evm: security.ima [ 3.682981] evm: security.capability [ 3.684365] evm: HMAC attrs: 0x1 [ 3.686675] rtc_cmos 00:05: setting system clock to 2025-11-01 02:05:14 UTC (1761962714) [ 3.693405] debug: unmapping init [mem 0xffffffff98c03000-0xffffffff98dfffff] [ 3.696339] debug: unmapping init [mem 0xffffffff97982000-0xffffffff97c58fff] [ 3.704557] Write protecting the kernel read-only data: 28672k [ 3.711741] debug: unmapping init [mem 0xffffffff96003000-0xffffffff961fffff] [ 3.714015] debug: unmapping init [mem 0xffffffff96914000-0xffffffff969fffff] [ 3.740510] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.747460] systemd[1]: Detected virtualization kvm. [ 3.749024] systemd[1]: Detected architecture x86-64. [ 3.750491] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.777722] systemd[1]: No hostname configured. [ 3.778982] systemd[1]: Set hostname to . [ 3.780329] random: systemd: uninitialized urandom read (16 bytes read) [ 3.782686] systemd[1]: Initializing machine ID from random generator. [ 3.822277] random: ln: uninitialized urandom read (6 bytes read) [ 3.908730] random: systemd: uninitialized urandom read (16 bytes read) [ 3.911760] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.920504] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 3.925324] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.481297] device-mapper: uevent: version 1.0.3 [ 4.483673] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.169488] virtio_net virtio0 ens2: renamed from eth0 [ 5.219554] random: fast init done [ 5.349875] scsi host0: ata_piix [ 5.412983] scsi host1: ata_piix [ 5.414447] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.416554] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.966105] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.192646] random: crng init done [ 10.193807] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.765934] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.934224] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.251891] SELinux: Disabled at runtime. [ 12.312219] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.319712] systemd[1]: Detected virtualization kvm. [ 12.321751] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.954647] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.960222] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.967792] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.972129] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.975370] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.993568] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.006567] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 13.108347] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.519775] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.858495] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.965226] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.095271] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.114759] EDAC sbridge: Ver: 1.1.2 [ 16.033956] Key type dns_resolver registered [ 16.341234] NFS: Registering the id_resolver key type [ 16.343059] Key type id_resolver registered [ 16.344665] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg203-server login: [ 38.220715] libcfs: loading out-of-tree module taints kernel. [ 38.232324] Key type ._llcrypt registered [ 38.233348] Key type .llcrypt registered [ 38.260447] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 45.435541] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 45.943892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 45.949885] alg: No test for adler32 (adler32-zlib) [ 46.923743] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 47.213547] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 48.824153] Key type lgssc registered [ 49.363508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 55.053728] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 55.385423] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 58.005118] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 60.381634] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 62.880907] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 68.069473] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 72.458364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 72.478264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 72.484332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 73.578629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 73.592504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 73.627498] Lustre: lustre-MDT0000: new disk, initializing [ 73.651980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 73.660463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 74.901858] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 79.946107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.974543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 80.005792] Lustre: 6458:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 80.019913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 80.021633] Lustre: Skipped 1 previous similar message [ 80.057882] Lustre: lustre-MDT0001: new disk, initializing [ 80.076172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 80.084006] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 80.086839] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 81.400602] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 83.665097] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 86.844385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.870035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 86.949572] Lustre: lustre-OST0000: new disk, initializing [ 86.951180] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 86.965678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 88.428970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 88.432763] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 88.459830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 88.818446] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 94.218761] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 94.245166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 94.286535] Lustre: lustre-OST0001: new disk, initializing [ 94.289502] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 94.308543] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 96.321642] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 98.796564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 98.801434] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 98.817830] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 102.154225] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 108.824124] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 115.712396] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing check_logdir /tmp/testlogs/ [ 117.160667] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing yml_node [ 118.530165] Lustre: DEBUG MARKER: Client: 2.16.59.55 [ 119.375474] Lustre: DEBUG MARKER: MDS: 2.16.59.55 [ 120.211469] Lustre: DEBUG MARKER: OSS: 2.16.59.55 [ 120.771110] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 31 22:07:12 EDT 2025 [ 126.539067] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 127.031573] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 139.745817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 139.746243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 139.751731] Lustre: Skipped 3 previous similar messages [ 139.755304] Lustre: Skipped 3 previous similar messages [ 144.864748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 144.867969] Lustre: Skipped 3 previous similar messages [ 144.898162] LustreError: 12117:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 144.924054] Lustre: server umount lustre-MDT0000 complete [ 146.106298] LustreError: 6449:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761962856 with bad export cookie 14357982196471847858 [ 146.107798] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 146.110033] LustreError: 6449:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 149.984769] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 149.985058] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 149.990124] Lustre: Skipped 1 previous similar message [ 152.032657] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 152.035529] Lustre: Skipped 2 previous similar messages [ 152.259564] LustreError: 12319:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 152.261311] LustreError: 12319:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 152.306097] Lustre: server umount lustre-MDT0001 complete [ 153.579564] LustreError: 12521:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 153.582446] LustreError: 12521:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 153.598409] Lustre: server umount lustre-OST0000 complete [ 154.924569] Lustre: server umount lustre-OST0001 complete [ 156.965993] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 160.033145] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 163.357306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 165.424224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 167.464489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 169.479544] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 171.746240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 171.769387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.853686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 171.865335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 171.900654] Lustre: lustre-MDT0000: new disk, initializing [ 171.927339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.933845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 173.186252] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 177.422626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 177.447878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 177.490309] Lustre: 14884:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 177.608823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 178.852217] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 181.884810] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 183.050662] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 185.318486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 185.346499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 185.445950] Lustre: lustre-OST0000: new disk, initializing [ 185.447488] Lustre: Skipped 1 previous similar message [ 185.449362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 185.451237] Lustre: Skipped 2 previous similar messages [ 185.485279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.488722] Lustre: Skipped 1 previous similar message [ 186.942951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 186.946780] Lustre: Skipped 1 previous similar message [ 186.948850] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 186.986175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 187.433835] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 190.558547] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 191.986294] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 192.077222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 193.349563] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 193.420304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 196.142142] LustreError: 17779:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 196.144711] LustreError: 17779:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 196.182475] Lustre: server umount lustre-OST0000 complete [ 198.113315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 198.117468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 198.122280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 202.210145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 207.330274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 207.332902] Lustre: Skipped 3 previous similar messages [ 207.336385] LustreError: 17979:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 207.337985] LustreError: 17979:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 207.367663] Lustre: server umount lustre-MDT0000 complete [ 208.675691] LustreError: 15658:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761962919 with bad export cookie 14357982196471849377 [ 208.681468] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 219.108220] LustreError: 18181:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 219.110297] LustreError: 18181:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 219.157221] Lustre: server umount lustre-MDT0001 complete [ 220.675978] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 22:08:52 (1761962932) [ 221.592946] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 222.223253] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 22:08:53 (1761962933) [ 224.396685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 224.557330] LustreError: 18851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 224.562039] LustreError: 18851:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 224.582215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 225.817120] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 228.587314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 230.034769] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 231.979860] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 233.136296] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 235.539704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 237.499711] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 239.319727] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 259.554549] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 259.557512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 259.561575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 260.583878] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 263.661945] LustreError: 21097:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 263.663834] LustreError: 21097:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 263.694819] Lustre: server umount lustre-OST0000 complete [ 271.127348] Lustre: server umount lustre-MDT0000 complete [ 272.337991] LustreError: 18831:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761962983 with bad export cookie 14357982196471850259 [ 272.340876] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 272.342217] LustreError: 18831:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 272.460304] Lustre: server umount lustre-MDT0001 complete [ 275.937745] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 276.917397] Key type lgssc unregistered [ 277.044339] LNet: 22025:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 277.046516] LNetError: 22025:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 277.053262] LNet: Removed LNI 192.168.202.103@tcp [ 277.398098] Key type .llcrypt unregistered [ 277.399147] Key type ._llcrypt unregistered [ 285.708263] Key type ._llcrypt registered [ 285.709391] Key type .llcrypt registered [ 285.735011] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 286.139658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 286.155556] alg: No test for adler32 (adler32-zlib) [ 287.009071] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 287.094296] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 288.672128] Key type lgssc registered [ 288.996464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 291.632487] loop: module loaded [ 291.654647] loop0: detected capacity change from 0 to 204800000 [ 291.777490] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 291.782710] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 291.892133] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 291.913553] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 294.139671] loop0: detected capacity change from 0 to 204800000 [ 294.207746] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 294.211352] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 294.216115] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 294.263368] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 295.908091] loop0: detected capacity change from 0 to 204800000 [ 297.737890] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 297.756783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 297.760539] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 298.842975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 298.861957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 298.902168] Lustre: lustre-MDT0000: new disk, initializing [ 298.925888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 298.933055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 298.942483] Lustre: 24122:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 466, rollback = 2 [ 298.945123] Lustre: 24122:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 298.947141] Lustre: 24122:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 2/466/0 [ 298.949383] Lustre: 24122:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 298.951532] Lustre: 24122:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 298.953487] Lustre: 24122:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 300.248035] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 303.778583] loop1: detected capacity change from 0 to 204800000 [ 305.639951] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 305.660493] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 305.756149] Lustre: lustre-OST07c6: new disk, initializing [ 305.758136] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 305.759914] Lustre: Skipped 1 previous similar message [ 305.784278] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 307.720689] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 312.323463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 312.326540] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 312.336515] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 315.394456] Lustre: 25147:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 315.398582] Lustre: 25147:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 1 previous similar message [ 315.402118] Lustre: 25147:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 315.404260] Lustre: 25147:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 315.407232] Lustre: 25147:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 315.410326] Lustre: 25147:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 315.412964] Lustre: 25147:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 315.416572] Lustre: 25147:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 315.418599] Lustre: 25147:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 315.421830] Lustre: 25147:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 315.424815] Lustre: 25147:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 315.428385] Lustre: 25147:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 331.824597] Lustre: 25147:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 497, rollback = 2 [ 331.826959] Lustre: 25147:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 331.828783] Lustre: 25147:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 331.830722] Lustre: 25147:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 331.832621] Lustre: 25147:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 331.834380] Lustre: 25147:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 335.308585] Lustre: 24147:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 497, rollback = 2 [ 335.311490] Lustre: 24147:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 1 previous similar message [ 335.313861] Lustre: 24147:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 335.316078] Lustre: 24147:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 335.318282] Lustre: 24147:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 335.320541] Lustre: 24147:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 335.322876] Lustre: 24147:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 335.325358] Lustre: 24147:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 335.327626] Lustre: 24147:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 335.330110] Lustre: 24147:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 335.332241] Lustre: 24147:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 335.334403] Lustre: 24147:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 341.835223] Lustre: 24147:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 497, rollback = 2 [ 341.838716] Lustre: 24147:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 3 previous similar messages [ 341.841236] Lustre: 24147:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 341.843688] Lustre: 24147:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 341.846208] Lustre: 24147:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 341.848835] Lustre: 24147:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 341.851448] Lustre: 24147:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 341.854391] Lustre: 24147:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 341.856989] Lustre: 24147:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 341.859360] Lustre: 24147:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 341.861959] Lustre: 24147:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 341.864904] Lustre: 24147:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 351.472720] Lustre: 24148:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 497, rollback = 2 [ 351.475692] Lustre: 24148:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 5 previous similar messages [ 351.477912] Lustre: 24148:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 351.480714] Lustre: 24148:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 351.483449] Lustre: 24148:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 351.485618] Lustre: 24148:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 351.488344] Lustre: 24148:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 351.490431] Lustre: 24148:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 351.492429] Lustre: 24148:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 351.494302] Lustre: 24148:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 351.496251] Lustre: 24148:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 351.498225] Lustre: 24148:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 399.330273] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 399.332436] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 399.336157] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 401.645538] LustreError: 26325:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 401.676178] Lustre: server umount lustre-OST07c6 complete [ 404.775340] LustreError: 26826:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 404.777244] LustreError: 26826:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 404.829711] Lustre: server umount lustre-MDT0000 complete [ 410.168233] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 411.115659] Key type lgssc unregistered [ 411.240380] LNet: 27650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 411.242883] LNetError: 27650:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 411.254434] LNet: Removed LNI 192.168.202.103@tcp [ 411.514841] Key type .llcrypt unregistered [ 411.515858] Key type ._llcrypt unregistered [ 418.523731] Key type ._llcrypt registered [ 418.524725] Key type .llcrypt registered [ 418.551118] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 418.894430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 418.914324] alg: No test for adler32 (adler32-zlib) [ 419.769924] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 419.846605] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 421.424103] Key type lgssc registered [ 421.773050] Lustre: Echo OBD driver; http://www.lustre.org/ [ 424.200691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 424.206510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.370335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.629597] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 429.419441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 429.556043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 430.786852] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 432.594740] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 433.727700] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 435.953331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 436.065563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 437.929543] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 439.765664] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 450.410162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 452.434867] LustreError: 31066:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 455.650368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.650641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 455.651496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 455.653743] Lustre: Skipped 1 previous similar message [ 455.655843] LustreError: Skipped 1 previous similar message [ 460.770104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 460.771844] Lustre: Skipped 2 previous similar messages [ 461.231090] LustreError: 31318:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 461.264210] Lustre: server umount lustre-OST0000 complete [ 465.377316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 465.380740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.386235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 468.800198] LustreError: 31520:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 468.802008] LustreError: 31520:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 468.831944] Lustre: server umount lustre-MDT0000 complete [ 470.144441] LustreError: 28850:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963180 with bad export cookie 17395774109982800286 [ 470.147919] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 470.149219] LustreError: 28850:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 470.213594] LustreError: 31722:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 470.215570] LustreError: 31722:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 470.269673] Lustre: server umount lustre-MDT0001 complete [ 473.947122] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 474.889493] Key type lgssc unregistered [ 475.003326] LNet: 32246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 475.005542] LNetError: 32246:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 475.018257] LNet: Removed LNI 192.168.202.103@tcp [ 475.293897] Key type .llcrypt unregistered [ 475.295045] Key type ._llcrypt unregistered [ 482.711132] Key type ._llcrypt registered [ 482.712256] Key type .llcrypt registered [ 482.740651] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 483.067922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 483.103066] alg: No test for adler32 (adler32-zlib) [ 483.952145] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 484.036818] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 485.616141] Key type lgssc registered [ 486.001086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 488.609418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 488.616582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.787863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.084427] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 493.900343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 494.022951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 495.254566] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 497.117085] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 498.288613] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 500.478646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 500.586926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 502.454349] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 504.251172] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 515.944912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 524.257101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 524.260956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 524.265281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 526.306926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 526.307171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 526.311407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.966161] LustreError: 35716:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 530.001762] Lustre: server umount lustre-OST0000 complete [ 531.426449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.427538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 531.431054] Lustre: Skipped 1 previous similar message [ 531.433743] Lustre: Skipped 1 previous similar message [ 536.545240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 536.547074] Lustre: Skipped 2 previous similar messages [ 537.405413] LustreError: 35918:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 537.407294] LustreError: 35918:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 537.436540] Lustre: server umount lustre-MDT0000 complete [ 538.688036] LustreError: 35070:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963249 with bad export cookie 12105962129386085471 [ 538.690827] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 538.690885] LustreError: 35070:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 538.762438] LustreError: 36121:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 538.764193] LustreError: 36121:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 538.814573] Lustre: server umount lustre-MDT0001 complete [ 542.326140] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 543.331715] Key type lgssc unregistered [ 543.455348] LNet: 36644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 543.457563] LNetError: 36644:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 543.464318] LNet: Removed LNI 192.168.202.103@tcp [ 543.760259] Key type .llcrypt unregistered [ 543.761474] Key type ._llcrypt unregistered [ 552.417502] Key type ._llcrypt registered [ 552.418538] Key type .llcrypt registered [ 552.444664] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 552.813428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 552.831374] alg: No test for adler32 (adler32-zlib) [ 553.676186] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 553.754810] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 555.336130] Key type lgssc registered [ 555.672096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 558.179043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 558.185147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 559.347486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 560.645193] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 563.512224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 563.637114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 564.948035] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 566.824720] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 567.994293] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 570.255413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 570.371955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 572.198795] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 573.968412] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 584.748495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 594.401197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 594.403623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.407850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 594.914349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.915655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 594.917748] Lustre: Skipped 1 previous similar message [ 594.922054] Lustre: Skipped 2 previous similar messages [ 598.144780] LustreError: 40256:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 598.175927] Lustre: server umount lustre-MDT0000 complete [ 599.481327] LustreError: 37990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963310 with bad export cookie 13596709195686472513 [ 599.484587] LustreError: 37990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 599.485575] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 599.999154] LustreError: 38007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.010723] LustreError: 38007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 600.011235] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.3@tcp (stopping) [ 600.034657] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 604.065844] LustreError: 39354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 604.066568] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 604.075041] LustreError: 39354:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 604.086724] Lustre: Skipped 1 previous similar message [ 605.118182] LustreError: 39354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 605.640467] LustreError: 40459:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 605.645183] LustreError: 40459:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 605.699297] Lustre: server umount lustre-MDT0001 complete [ 623.597674] LustreError: 40682:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 623.599995] LustreError: 40682:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 623.637889] Lustre: server umount lustre-OST0000 complete [ 628.905180] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 629.883361] Key type lgssc unregistered [ 630.013365] LNet: 41510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.016996] LNetError: 41510:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.026318] LNet: Removed LNI 192.168.202.103@tcp [ 630.358255] Key type .llcrypt unregistered [ 630.359802] Key type ._llcrypt unregistered [ 639.480929] Key type ._llcrypt registered [ 639.482564] Key type .llcrypt registered [ 639.514451] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 639.906886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 639.913825] alg: No test for adler32 (adler32-zlib) [ 640.769673] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 640.853258] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 642.432217] Key type lgssc registered [ 642.781801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 645.283184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 645.289556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.446691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 647.736594] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 650.656473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 650.800822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 652.016682] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 653.764769] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 654.923629] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 657.133303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 657.271621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 659.155669] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 660.896793] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 671.592119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 680.929082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 680.933047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 680.939883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 681.953929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 681.954978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 681.955763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 681.956926] Lustre: Skipped 1 previous similar message [ 684.598110] LustreError: 45124:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 684.638730] Lustre: server umount lustre-OST0000 complete [ 701.921230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 701.923956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 701.928903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 703.552049] LustreError: 45500:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 703.554513] LustreError: 45500:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 703.585454] Lustre: server umount lustre-MDT0000 complete [ 704.838390] LustreError: 42857:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963415 with bad export cookie 3099571213525398366 [ 704.841630] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 704.841891] LustreError: 42857:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 719.328198] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 719.363953] LustreError: 45701:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 719.367136] LustreError: 45701:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 719.430770] Lustre: server umount lustre-MDT0001 complete [ 723.219959] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 724.290503] Key type lgssc unregistered [ 724.420441] LNet: 46228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 724.423585] LNetError: 46228:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 724.433409] LNet: Removed LNI 192.168.202.103@tcp [ 724.749759] Key type .llcrypt unregistered [ 724.751081] Key type ._llcrypt unregistered [ 733.432986] Key type ._llcrypt registered [ 733.434496] Key type .llcrypt registered [ 733.462192] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 733.884358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 733.890904] alg: No test for adler32 (adler32-zlib) [ 734.742472] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 734.825533] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 736.408134] Key type lgssc registered [ 736.784117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 739.321060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 739.327872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 740.483780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 741.778328] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 744.603164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 744.733590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 746.029553] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 747.787698] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 748.920616] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 751.093949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 751.213345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 752.967540] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 754.706709] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 765.546992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 772.531062] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 773.598054] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 784.272907] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-3122h18m15s [ 784.769681] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 785.860575] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 785.888759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 785.891801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 786.877829] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 786.881447] Lustre: Skipped 3 previous similar messages [ 791.550578] LustreError: 50157:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 791.575326] Lustre: server umount lustre-MDT0000 complete [ 792.649722] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 793.021371] LustreError: 47588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 793.025670] LustreError: 47588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 794.842681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 794.881373] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 794.972837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 794.993475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 796.011496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 796.281602] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 797.417476] Lustre: DEBUG MARKER: Wait for df (35483) ... [ 797.912042] Lustre: DEBUG MARKER: done [ 800.035056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 800.036151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 800.038424] Lustre: Skipped 3 previous similar messages [ 800.040111] Lustre: Skipped 3 previous similar messages [ 800.943101] LustreError: 51197:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 800.945971] LustreError: 51197:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 800.979284] Lustre: server umount lustre-OST0000 complete [ 804.349125] LustreError: 51398:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 804.350867] LustreError: 51398:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 804.373615] Lustre: server umount lustre-MDT0000 complete [ 805.345653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 805.346318] LustreError: 47590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 805.349562] Lustre: Skipped 1 previous similar message [ 805.356560] LustreError: 47590:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 805.634336] LustreError: 47574:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963516 with bad export cookie 4080545648604103677 [ 805.637527] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 805.640332] LustreError: 47574:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 811.774765] LustreError: 51602:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 811.777987] LustreError: 51602:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 811.829785] Lustre: server umount lustre-MDT0001 complete [ 815.434353] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 816.491976] Key type lgssc unregistered [ 816.632368] LNet: 52126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.634949] LNetError: 52126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.643332] LNet: Removed LNI 192.168.202.103@tcp [ 816.921893] Key type .llcrypt unregistered [ 816.923029] Key type ._llcrypt unregistered [ 828.555720] Key type ._llcrypt registered [ 828.556866] Key type .llcrypt registered [ 828.581363] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 834.842663] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 835.178686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 835.201068] alg: No test for adler32 (adler32-zlib) [ 836.053705] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 836.135758] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 837.720228] Key type lgssc registered [ 838.210315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 841.307319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 843.498944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 845.865204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 847.938768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 850.555777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 850.579780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 850.586761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.681756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 851.693625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 851.735839] Lustre: lustre-MDT0000: new disk, initializing [ 851.760146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 851.767089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 853.054367] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 857.299751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 857.323677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 857.358093] Lustre: 56032:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 857.382467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 857.384116] Lustre: Skipped 1 previous similar message [ 857.420386] Lustre: lustre-MDT0001: new disk, initializing [ 857.451166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 857.462780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 857.467085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 858.840870] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 861.918735] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 863.131924] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 865.410175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 865.436023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 865.556424] Lustre: lustre-OST0000: new disk, initializing [ 865.559070] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 865.593780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 867.608338] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 870.625150] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 872.072400] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 873.471678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 873.476488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 873.510617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 874.189957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 875.445341] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 875.505796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 878.562483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 878.563258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 878.565309] LustreError: Skipped 1 previous similar message [ 878.570069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 878.572723] Lustre: Skipped 1 previous similar message [ 882.285926] LustreError: 58940:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 882.333701] Lustre: server umount lustre-OST0000 complete [ 883.681360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 883.687119] Lustre: Skipped 1 previous similar message [ 883.690136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 888.289633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 888.296227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 888.305291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 889.801616] LustreError: 59141:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 889.806193] LustreError: 59141:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 889.840801] Lustre: server umount lustre-MDT0000 complete [ 891.201325] LustreError: 56024:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963602 with bad export cookie 13568054080421612579 [ 891.204460] LustreError: 56024:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 891.205138] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 891.267977] LustreError: 59343:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 891.270531] LustreError: 59343:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 891.320764] Lustre: server umount lustre-MDT0001 complete [ 894.127551] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 22:20:05 (1761963605) [ 896.594388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 896.826269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 898.098809] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 901.242306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 901.381714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 902.761185] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 904.546631] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 905.843358] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 908.303234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 910.409517] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 912.148970] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 924.960717] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-3122h20m36s [ 925.575649] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 928.005477] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 928.824498] Lustre: DEBUG MARKER: Creating a test file and stat it [ 928.839950] LustreError: 60008:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 949.194801] Lustre: lustre-MDT0000: Client 5527bdaf-c6b7-432f-8581-b241455a8764 (at 192.168.202.3@tcp) reconnecting [ 968.880129] LustreError: 60008:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 969.609836] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 970.397096] Lustre: DEBUG MARKER: done [ 973.284392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 973.284494] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 973.285448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 973.285452] Lustre: Skipped 1 previous similar message [ 973.286393] LustreError: Skipped 1 previous similar message [ 973.294549] Lustre: Skipped 1 previous similar message [ 978.094568] LustreError: 63015:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 978.096305] LustreError: 63015:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 978.129586] Lustre: server umount lustre-OST0000 complete [ 983.523331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 983.523417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.524239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 983.524247] Lustre: Skipped 1 previous similar message [ 983.531752] Lustre: Skipped 1 previous similar message [ 985.468993] LustreError: 63217:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 985.470936] LustreError: 63217:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 985.497651] Lustre: server umount lustre-MDT0000 complete [ 986.736268] LustreError: 61618:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963697 with bad export cookie 13568054080421613447 [ 986.739098] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 986.741389] LustreError: 61618:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 986.851540] Lustre: server umount lustre-MDT0001 complete [ 990.306578] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 991.457967] Key type lgssc unregistered [ 991.598381] LNet: 63945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.600658] LNetError: 63945:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.612529] LNet: Removed LNI 192.168.202.103@tcp [ 991.956472] Key type .llcrypt unregistered [ 991.957708] Key type ._llcrypt unregistered [ 1002.948658] Key type ._llcrypt registered [ 1002.949929] Key type .llcrypt registered [ 1002.977536] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 1009.109035] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1009.625784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1009.647754] alg: No test for adler32 (adler32-zlib) [ 1010.507306] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1010.594793] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1012.176136] Key type lgssc registered [ 1012.553204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1015.353265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1017.440629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1019.367382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1021.498653] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1023.887897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1023.902779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1023.907524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1025.007411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1025.020183] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1025.064442] Lustre: lustre-MDT0000: new disk, initializing [ 1025.093264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1025.100569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1026.468608] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1030.715984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1030.737889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1030.777376] Lustre: 67851:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1030.799260] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1030.800898] Lustre: Skipped 1 previous similar message [ 1030.835961] Lustre: lustre-MDT0001: new disk, initializing [ 1030.869173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1030.881533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1030.885577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1032.139102] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1035.402139] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1036.664579] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1038.939250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1038.968064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1039.084647] Lustre: lustre-OST0000: new disk, initializing [ 1039.086548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1039.121198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.061724] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1042.177979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1042.181774] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1042.221606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1044.146885] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1045.503173] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1045.574387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1046.878479] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1046.944717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1052.644616] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1052.646080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1052.647965] LustreError: Skipped 1 previous similar message [ 1052.648544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1052.651993] Lustre: Skipped 1 previous similar message [ 1052.656182] Lustre: Skipped 1 previous similar message [ 1053.679112] LustreError: 70743:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1053.715417] Lustre: server umount lustre-OST0000 complete [ 1056.736931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1056.739120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.743652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1057.761520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1057.762776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.015739] LustreError: 70945:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1061.021431] LustreError: 70945:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1061.064696] Lustre: server umount lustre-MDT0000 complete [ 1062.522334] LustreError: 68623:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963773 with bad export cookie 4549728366187889510 [ 1062.527255] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1062.530139] LustreError: 68623:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1062.613229] LustreError: 71148:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1062.615797] LustreError: 71148:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1062.680714] Lustre: server umount lustre-MDT0001 complete [ 1066.244985] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 22:22:57 (1761963777) [ 1066.896654] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1067.640782] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 22:22:59 (1761963779) [ 1071.977695] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1073.549921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.732130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1074.051277] Lustre: Failing over lustre-MDT0000 [ 1074.052731] LustreError: 72213:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1074.055863] LustreError: 72213:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1074.058706] LustreError: 72213:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1074.129496] LustreError: 72264:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1074.133992] LustreError: 72264:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1074.182310] Lustre: server umount lustre-MDT0000 complete [ 1076.206180] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 22:23:07 (1761963787) [ 1078.449463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.678202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1080.022329] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1083.020054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1084.395666] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1086.478610] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1087.774490] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1090.669680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1090.832993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.837131] Lustre: Skipped 1 previous similar message [ 1093.258450] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1095.520471] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1107.899820] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1114.081936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1114.087946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.094894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1114.098433] Lustre: Skipped 1 previous similar message [ 1115.784549] LustreError: 75173:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1115.789049] LustreError: 75173:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1115.829910] Lustre: server umount lustre-MDT0000 complete [ 1117.280518] LustreError: 72811:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963828 with bad export cookie 4549728366187890651 [ 1117.285582] LustreError: 72811:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1117.286331] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1117.426935] Lustre: server umount lustre-MDT0001 complete [ 1118.873090] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1120.639389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1124.293300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1127.841373] LustreError: 66068:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000089b3912d x1847552870770688/t0(0) o250->MGC192.168.202.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1128.016668] LustreError: 74424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1128.017501] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.032145] LustreError: 74424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1128.035906] Lustre: Skipped 2 previous similar messages [ 1128.076433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.111629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1129.450841] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1132.685680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1134.500421] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1134.954230] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1135.968120] Lustre: 66070:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761963830/real 1761963830] req@00000000111005bf x1847552870770304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761963846 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1136.443565] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1137.736302] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1139.024127] Lustre: 66070:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761963833/real 1761963833] req@00000000453a4eb8 x1847552870770432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761963849 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1140.197840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1140.200154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1140.207213] Lustre: Skipped 1 previous similar message [ 1140.211048] Lustre: Skipped 2 previous similar messages [ 1141.216112] Lustre: 66072:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761963835/real 1761963835] req@000000005491f345 x1847552870770560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761963851 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1143.264201] Lustre: 66071:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761963838/real 1761963838] req@0000000083a19c59 x1847552870770816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761963854 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1146.182206] LustreError: 77623:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1146.184868] LustreError: 77623:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1146.226216] Lustre: server umount lustre-MDT0000 complete [ 1147.728921] LustreError: 72813:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963858 with bad export cookie 4549728366187892541 [ 1147.735556] LustreError: 72813:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1147.735783] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1147.881206] Lustre: server umount lustre-MDT0001 complete [ 1150.316793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1153.519831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1157.600891] LustreError: 66068:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000966700ef x1847552870790016/t0(0) o250->MGC192.168.202.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1157.740758] LustreError: 74423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1157.793980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1159.143640] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1162.295122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1162.388875] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.393108] Lustre: Skipped 1 previous similar message [ 1162.419557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1162.421595] Lustre: Skipped 2 previous similar messages [ 1164.052224] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1166.305160] Lustre: 66070:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761963861/real 1761963861] req@00000000c74fe4cd x1847552870789632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761963877 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1166.534333] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1167.850764] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1167.854721] Lustre: Skipped 1 previous similar message [ 1168.228428] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1171.493342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1171.496470] Lustre: Skipped 3 previous similar messages [ 1176.577805] LustreError: 80063:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1176.580040] LustreError: 80063:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1176.609620] Lustre: server umount lustre-MDT0000 complete [ 1177.924442] LustreError: 75176:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963888 with bad export cookie 4549728366187893913 [ 1177.928622] LustreError: 75176:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1177.929123] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1178.058789] Lustre: server umount lustre-MDT0001 complete [ 1180.420894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1181.943915] Lustre: DEBUG MARKER: files compared the same [ 1193.148803] Lustre: server umount lustre-OST0000 complete [ 1199.655421] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1200.718107] Key type lgssc unregistered [ 1200.865565] LNet: 81686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1200.868598] LNetError: 81686:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1200.875549] LNet: Removed LNI 192.168.202.103@tcp [ 1201.251059] Key type .llcrypt unregistered [ 1201.252575] Key type ._llcrypt unregistered [ 1210.673607] Key type ._llcrypt registered [ 1210.674796] Key type .llcrypt registered [ 1210.713909] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1211.020131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1211.078592] alg: No test for adler32 (adler32-zlib) [ 1211.935578] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1212.029646] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1213.616254] Key type lgssc registered [ 1214.071430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1216.946545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1216.952244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1218.145103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1219.448616] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug +malloc all [ 1222.315053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1222.408829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1223.700837] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug +malloc all [ 1225.429719] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1226.594537] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1228.766670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1228.866691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1230.653049] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug +malloc all [ 1232.409270] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1242.211777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1247.712976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1247.713778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.714132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1247.719854] Lustre: Skipped 1 previous similar message [ 1251.691565] LustreError: 85293:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1251.721323] Lustre: server umount lustre-OST0000 complete [ 1253.345077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1253.349618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.356853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1253.360676] Lustre: Skipped 1 previous similar message [ 1257.952656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1257.952891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1259.069080] LustreError: 85495:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1259.070844] LustreError: 85495:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1259.092824] Lustre: server umount lustre-MDT0000 complete [ 1260.387756] LustreError: 83033:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963971 with bad export cookie 11096756991850059850 [ 1260.389142] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1260.390445] LustreError: 83033:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1260.463106] LustreError: 85698:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1260.464590] LustreError: 85698:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1260.512841] Lustre: server umount lustre-MDT0001 complete [ 1264.341650] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1265.536422] Key type lgssc unregistered [ 1265.694468] LNet: 86222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1265.698604] LNetError: 86222:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1265.710406] LNet: Removed LNI 192.168.202.103@tcp [ 1266.148752] Key type .llcrypt unregistered [ 1266.151665] Key type ._llcrypt unregistered [ 1276.190727] Key type ._llcrypt registered [ 1276.192215] Key type .llcrypt registered [ 1276.220517] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1276.606850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1276.621590] alg: No test for adler32 (adler32-zlib) [ 1277.477622] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1277.563228] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1279.160217] Key type lgssc registered [ 1279.726641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1282.750116] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1282.759711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1298.208369] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1303.328342] LustreError: 87560:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 1329.952281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1332.166966] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1334.155979] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1337.219195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1337.290838] Lustre: *** cfs_fail_loc=706, val=0*** [ 1337.514067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1337.556831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1339.233705] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1342.875885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1342.947143] LustreError: 87587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1342.954673] LustreError: 87587:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1343.094079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1344.823827] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1346.859910] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1348.178651] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1350.627332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1350.628728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1350.630594] Lustre: Skipped 1 previous similar message [ 1355.121130] LustreError: 89864:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1355.163519] Lustre: server umount lustre-OST0000 complete [ 1358.305387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1358.309059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.314915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1358.317635] Lustre: Skipped 1 previous similar message [ 1360.865510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1360.866050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.758222] LustreError: 90067:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1362.761473] LustreError: 90067:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1362.797314] Lustre: server umount lustre-MDT0000 complete [ 1364.253296] LustreError: 87568:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964075 with bad export cookie 207217348079454945 [ 1364.257286] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1364.258556] LustreError: 87568:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1364.339417] LustreError: 90270:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1364.342000] LustreError: 90270:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1364.392704] Lustre: server umount lustre-MDT0001 complete [ 1368.590938] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1369.957985] Key type lgssc unregistered [ 1370.084594] LNet: 90794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1370.088131] LNetError: 90794:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1370.101470] LNet: Removed LNI 192.168.202.103@tcp [ 1370.498561] Key type .llcrypt unregistered [ 1370.500096] Key type ._llcrypt unregistered [ 1381.522322] Key type ._llcrypt registered [ 1381.523697] Key type .llcrypt registered [ 1381.552817] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1381.935713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1381.964905] alg: No test for adler32 (adler32-zlib) [ 1382.834099] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1382.925765] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1384.504204] Key type lgssc registered [ 1384.989626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1388.003354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1388.010059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.373489] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1393.986216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1394.275504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1395.906614] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1399.778894] LustreError: 92822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1399.789724] LustreError: 92822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1399.938522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1400.151520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1402.498399] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1404.258203] LustreError: 92826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1404.263559] LustreError: 92826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1405.696981] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1405.702685] Lustre: MGS: Client 757ad361-f0da-4e81-a23f-5a185307799c (at 0@lo) reconnecting [ 1405.807362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1405.832371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1407.071169] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1416.165863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.167073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1416.169687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1416.174322] Lustre: Skipped 1 previous similar message [ 1421.285899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1421.290551] Lustre: Skipped 2 previous similar messages [ 1421.557174] LustreError: 94763:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1421.605429] Lustre: server umount lustre-OST0000 complete [ 1426.402963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1426.404215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1426.415782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1426.419192] Lustre: Skipped 1 previous similar message [ 1429.263228] LustreError: 94965:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1429.268214] LustreError: 94965:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1429.281908] Lustre: server umount lustre-MDT0000 complete [ 1430.872608] LustreError: 95167:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1430.875439] LustreError: 95167:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1430.921658] Lustre: server umount lustre-MDT0001 complete [ 1432.519072] Lustre: server umount MGS complete [ 1437.223449] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 22:29:08 (1761964148) [ 1449.717360] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 1453.782308] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1457.283230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1459.821901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1462.371697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1464.518817] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1467.272954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1467.305803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1468.677763] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1473.568914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1473.609972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1473.739311] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1473.756201] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1473.795273] Lustre: lustre-MDT0001: new disk, initializing [ 1473.822034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1473.832581] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1473.834647] Lustre: 99855:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1473.836940] LustreError: 99855:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1475.344091] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1481.001781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1481.046147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1481.195967] Lustre: lustre-OST0000: new disk, initializing [ 1481.199333] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1481.232544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1483.820794] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1487.939293] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1490.665432] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1490.673263] Lustre: MGS: Client c8628a27-bb15-4b63-a55c-df8f9b8cde36 (at 0@lo) reconnecting [ 1490.700935] Lustre: 99147:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1490.724417] LustreError: 99827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1490.755806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1490.767993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 1490.817904] Lustre: lustre-MDT0000: new disk, initializing [ 1490.877897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1490.892065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1492.522328] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1496.048830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1496.055233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1496.932749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1497.062338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1517.537823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1517.538427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1517.541469] LustreError: Skipped 1 previous similar message [ 1517.547279] Lustre: Skipped 1 previous similar message [ 1517.548050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1517.553893] Lustre: Skipped 2 previous similar messages [ 1526.761584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1526.765874] Lustre: Skipped 3 previous similar messages [ 1531.360271] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1531.387183] LustreError: 102071:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1531.391679] LustreError: 102071:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1531.458211] Lustre: server umount lustre-OST0000 complete [ 1536.995788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1536.996173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1537.003491] Lustre: Skipped 2 previous similar messages [ 1539.478240] LustreError: 102275:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1539.482434] LustreError: 102275:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1539.496356] Lustre: server umount lustre-MDT0000 complete [ 1547.474313] Lustre: server umount lustre-MDT0001 complete [ 1549.196100] LustreError: 102679:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1549.199508] LustreError: 102679:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1549.234401] Lustre: server umount MGS complete [ 1554.402672] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 22:31:05 (1761964265) [ 1562.503166] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1563.803668] Key type lgssc unregistered [ 1563.972706] LNet: 104006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1563.977272] LNetError: 104006:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1563.985600] LNet: Removed LNI 192.168.202.103@tcp [ 1564.423254] Key type .llcrypt unregistered [ 1564.425286] Key type ._llcrypt unregistered [ 1571.960479] Key type ._llcrypt registered [ 1571.961619] Key type .llcrypt registered [ 1571.995504] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1572.453650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1572.501397] alg: No test for adler32 (adler32-zlib) [ 1573.358401] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1573.452116] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1575.032182] Key type lgssc registered [ 1575.680364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1579.248859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1579.259555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1580.458629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1582.132312] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1586.297017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1586.449941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1587.946824] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1590.366825] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1591.984748] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1594.934043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1595.118058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1597.509667] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1599.990673] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1605.363560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 1610.724320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1610.724649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1610.725607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1610.727946] Lustre: Skipped 1 previous similar message [ 1614.323189] LustreError: 107404:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1614.367746] Lustre: server umount lustre-OST0000 complete [ 1615.842752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1615.843428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1615.847193] Lustre: Skipped 1 previous similar message [ 1615.848761] Lustre: Skipped 1 previous similar message [ 1620.962209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1620.966122] Lustre: Skipped 2 previous similar messages [ 1626.081145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1626.082677] Lustre: Skipped 1 previous similar message [ 1630.176253] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1630.224185] LustreError: 107605:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1630.228018] LustreError: 107605:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1630.258747] Lustre: server umount lustre-MDT0000 complete [ 1631.202188] LustreError: 106326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1631.207223] LustreError: 106326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1631.940150] LustreError: 105157:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964342 with bad export cookie 17549749205930302179 [ 1631.946994] LustreError: 105157:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1631.949257] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1632.037828] LustreError: 107810:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1632.041460] LustreError: 107810:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1632.111485] Lustre: server umount lustre-MDT0001 complete [ 1636.410314] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1637.707081] Key type lgssc unregistered [ 1637.855664] LNet: 108334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1637.857992] LNetError: 108334:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1637.864474] LNet: Removed LNI 192.168.202.103@tcp [ 1638.257504] Key type .llcrypt unregistered [ 1638.259618] Key type ._llcrypt unregistered [ 1646.049648] Key type ._llcrypt registered [ 1646.050660] Key type .llcrypt registered [ 1646.091580] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1646.595367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1646.608105] alg: No test for adler32 (adler32-zlib) [ 1647.474851] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1647.578354] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1649.184246] Key type lgssc registered [ 1649.819893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1652.835410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1652.840215] LustreError: 109365:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1652.850107] LustreError: 109367:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1652.854531] LustreError: 109365:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4990 [ 1652.859175] LustreError: 109365:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1652.866174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1652.866607] LustreError: 109365:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1654.128218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.657463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1657.817446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1659.580268] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1661.706694] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing lsmod [ 1663.148067] LustreError: 110529:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1663.148505] LustreError: 110530:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1663.157086] LustreError: 110529:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 1663.161042] LustreError: 110529:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1663.166594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1663.167989] LustreError: 110529:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1663.320530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1664.365469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:65) [ 1667.748159] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1667.750912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1667.754273] Lustre: Skipped 2 previous similar messages [ 1667.757400] Lustre: Skipped 1 previous similar message [ 1670.803254] LustreError: 110717:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1670.842344] Lustre: server umount lustre-MDT0000 complete [ 1672.610166] LustreError: 109387:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964383 with bad export cookie 11971304747487027018 [ 1672.616673] LustreError: 109387:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1672.617903] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1673.186300] LustreError: 109846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1673.186932] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.196161] LustreError: 109846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1673.204428] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1673.888309] LustreError: 109839:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d92a5b83 x1847553538672512/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1673.907615] LustreError: 110921:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1673.911759] LustreError: 110921:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1673.989955] Lustre: server umount lustre-MDT0001 complete [ 1675.893402] LustreError: 111123:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1675.897471] LustreError: 111123:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1675.956621] Lustre: server umount lustre-OST0000 complete [ 1680.202976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1680.426455] LustreError: 111599:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1680.462706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1682.211677] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1685.475148] LustreError: 111617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1686.300235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1686.472240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1688.203512] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1690.639377] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1692.246374] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1695.034944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1695.181354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1697.292513] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1699.720344] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1705.452873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:97) [ 1715.686161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1715.691553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1715.692829] Lustre: Skipped 1 previous similar message [ 1715.699868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1717.875178] LustreError: 113846:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1717.878651] LustreError: 113846:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1717.931527] Lustre: server umount lustre-OST0000 complete [ 1720.804241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1720.805623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1720.809635] Lustre: Skipped 1 previous similar message [ 1720.815566] Lustre: Skipped 2 previous similar messages [ 1725.511573] LustreError: 114047:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1725.514559] LustreError: 114047:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1725.553114] Lustre: server umount lustre-MDT0000 complete [ 1725.922627] LustreError: 111617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1725.929921] LustreError: 111617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1727.255240] LustreError: 111625:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964438 with bad export cookie 11971304747487027935 [ 1727.261557] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1727.262468] LustreError: 111625:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1727.398949] Lustre: server umount lustre-MDT0001 complete [ 1731.478985] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1732.939852] Key type lgssc unregistered [ 1733.110604] LNet: 114773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1733.114658] LNetError: 114773:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.127698] LNet: Removed LNI 192.168.202.103@tcp [ 1733.566596] Key type .llcrypt unregistered [ 1733.568333] Key type ._llcrypt unregistered [ 1744.783387] Key type ._llcrypt registered [ 1744.784590] Key type .llcrypt registered [ 1744.819852] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1745.287482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1745.353285] alg: No test for adler32 (adler32-zlib) [ 1746.220888] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1746.309765] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1747.888142] Key type lgssc registered [ 1748.348223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1751.320923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1751.327773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1752.499848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1754.046505] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1757.976545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1758.175786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1759.832739] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1762.365404] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1763.891788] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1767.054609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1767.220152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1769.595258] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1771.908892] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1777.456340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1779.720567] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1780.625611] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1782.754884] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1782.759625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.761523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1782.768560] Lustre: Skipped 1 previous similar message [ 1783.987197] LustreError: 118470:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1784.048031] Lustre: server umount lustre-OST0000 complete [ 1787.875477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1787.876641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1787.883114] Lustre: Skipped 1 previous similar message [ 1787.889833] Lustre: Skipped 2 previous similar messages [ 1787.983099] LustreError: 118675:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1787.987212] LustreError: 118675:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1788.020347] Lustre: server umount lustre-MDT0000 complete [ 1789.796160] LustreError: 118322:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964500 with bad export cookie 8518461293115566997 [ 1789.804475] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1789.807162] LustreError: 118322:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1789.896173] LustreError: 118875:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1789.898973] LustreError: 118875:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1789.958201] Lustre: server umount lustre-MDT0001 complete [ 1794.112970] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1795.137419] Key type lgssc unregistered [ 1795.270682] LNet: 119399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.274397] LNetError: 119399:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1795.288508] LNet: Removed LNI 192.168.202.103@tcp [ 1795.731835] Key type .llcrypt unregistered [ 1795.734833] Key type ._llcrypt unregistered [ 1804.525883] Key type ._llcrypt registered [ 1804.527206] Key type .llcrypt registered [ 1804.560612] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1804.944817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1804.993896] alg: No test for adler32 (adler32-zlib) [ 1805.850792] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1805.953821] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1807.544202] Key type lgssc registered [ 1808.074126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1811.595599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1811.602531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1812.804745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1814.337757] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1818.316719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1818.517726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1820.290214] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1823.006553] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1824.837850] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1828.057698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1828.200263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.541536] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1833.034330] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1837.421379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1842.661491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1842.664525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1842.664771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1842.669342] Lustre: Skipped 1 previous similar message [ 1842.680050] Lustre: Skipped 1 previous similar message [ 1843.635136] LustreError: 122796:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1843.689128] Lustre: server umount lustre-OST0000 complete [ 1847.698507] LustreError: 122998:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1847.702280] LustreError: 122998:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1847.736953] Lustre: server umount lustre-MDT0000 complete [ 1847.777834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1847.778771] LustreError: 121672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1847.782944] Lustre: Skipped 1 previous similar message [ 1847.792824] LustreError: 121672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1849.543826] LustreError: 120553:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964560 with bad export cookie 1236335705553020197 [ 1849.550620] LustreError: 120553:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1849.550965] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1849.652643] LustreError: 123201:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1849.655859] LustreError: 123201:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1849.713987] Lustre: server umount lustre-MDT0001 complete [ 1853.796796] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1854.829310] Key type lgssc unregistered [ 1854.951404] LNet: 123726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1854.954405] LNetError: 123726:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1854.969341] LNet: Removed LNI 192.168.202.103@tcp [ 1855.321680] Key type .llcrypt unregistered [ 1855.324346] Key type ._llcrypt unregistered [ 1866.603755] Key type ._llcrypt registered [ 1866.604977] Key type .llcrypt registered [ 1866.649722] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1867.039179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1867.075191] alg: No test for adler32 (adler32-zlib) [ 1867.927889] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1868.009264] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1869.592205] Key type lgssc registered [ 1870.204644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1874.192264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1874.206156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1875.411182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1877.081760] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1881.323162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1881.507042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1883.322297] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1885.639773] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1887.188435] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1890.358599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1890.492250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1893.069515] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1895.666683] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1900.791831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1903.888694] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1905.740172] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1905.743986] Lustre: Skipped 2 previous similar messages [ 1906.846708] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 1906.865774] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1907.956554] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1907.960502] Lustre: Skipped 2 previous similar messages [ 1916.939117] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1916.942540] Lustre: Skipped 1 previous similar message [ 1919.492882] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.3@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1919.499132] Lustre: Skipped 2 previous similar messages [ 1926.666870] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.3@tcp 192.168.0.[2,10]@tcp [ 1931.447364] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.3/24@tcp in log lustre-MDT0000 [ 1931.452497] Lustre: Skipped 2 previous similar messages [ 1936.395785] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.3/24@tcp [ 1936.398632] Lustre: Skipped 1 previous similar message [ 1941.996201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1941.998627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1942.004450] Lustre: Skipped 1 previous similar message [ 1942.011589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1945.780958] LustreError: 129065:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1945.839697] Lustre: server umount lustre-OST0000 complete [ 1948.129836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1948.136335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.148898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1948.152990] Lustre: Skipped 1 previous similar message [ 1952.227246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1952.227921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.809436] LustreError: 129267:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1953.811384] LustreError: 129267:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1953.845803] Lustre: server umount lustre-MDT0000 complete [ 1955.593662] LustreError: 128716:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964666 with bad export cookie 1131458807387659638 [ 1955.599620] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1955.599918] LustreError: 128716:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1955.689724] LustreError: 129469:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1955.692833] LustreError: 129469:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1955.742808] Lustre: server umount lustre-MDT0001 complete [ 1959.828303] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1961.101073] Key type lgssc unregistered [ 1961.268693] LNet: 129993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1961.273449] LNetError: 129993:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1961.288609] LNet: Removed LNI 192.168.202.103@tcp [ 1961.762237] Key type .llcrypt unregistered [ 1961.764344] Key type ._llcrypt unregistered [ 1975.749756] Key type ._llcrypt registered [ 1975.751403] Key type .llcrypt registered [ 1975.787845] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1976.075069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1976.081280] alg: No test for adler32 (adler32-zlib) [ 1976.931972] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1977.019536] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1978.600181] Key type lgssc registered [ 1979.180406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1982.953950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1982.965105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1984.149026] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1984.154656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.156447] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.3/24@tcp [ 1985.821858] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1990.069215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1990.211362] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1990.223268] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.3/24@tcp [ 1990.247525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1992.035738] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1994.771097] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1996.618280] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2000.094203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2000.306497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.999166] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2005.678236] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2008.497287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:201 to 0x280000401:225) [ 2016.736764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2016.738925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2016.742555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2018.788701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2018.790083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2018.793568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.804867] Lustre: Skipped 1 previous similar message [ 2020.592096] LustreError: 133790:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2020.634896] Lustre: server umount lustre-OST0000 complete [ 2023.907776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.908814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2023.914886] Lustre: Skipped 1 previous similar message [ 2028.542495] LustreError: 133992:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2028.544319] LustreError: 133992:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2028.569507] Lustre: server umount lustre-MDT0000 complete [ 2029.025784] LustreError: 131510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2029.034384] LustreError: 131510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2030.254728] LustreError: 131489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964741 with bad export cookie 17901649715737697126 [ 2030.261448] LustreError: 131489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2030.261990] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2030.352513] LustreError: 134193:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2030.355996] LustreError: 134193:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2030.406461] Lustre: server umount lustre-MDT0001 complete [ 2035.283782] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2036.893339] Key type lgssc unregistered [ 2037.093565] LNet: 134718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.098950] LNetError: 134718:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2037.110763] LNet: Removed LNI 192.168.202.103@tcp [ 2037.599751] Key type .llcrypt unregistered [ 2037.601914] Key type ._llcrypt unregistered [ 2049.092276] Key type ._llcrypt registered [ 2049.093394] Key type .llcrypt registered [ 2049.123576] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 2056.807334] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2057.186139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2057.243045] alg: No test for adler32 (adler32-zlib) [ 2058.145918] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 2058.242101] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2059.840180] Key type lgssc registered [ 2060.425340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2064.547907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2067.626112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2070.621167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2073.615320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2076.664823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2076.684671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2076.688509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.776301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2077.790874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2077.832841] Lustre: lustre-MDT0000: new disk, initializing [ 2077.859372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2077.866194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2079.547706] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2084.889735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2084.925685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2084.978601] Lustre: 138051:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2085.018063] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2085.021959] Lustre: Skipped 1 previous similar message [ 2085.076416] Lustre: lustre-MDT0001: new disk, initializing [ 2085.135945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2085.158117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2085.165102] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2087.082986] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2091.097464] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2092.771789] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2096.215217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2096.244084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2096.349260] Lustre: lustre-OST0000: new disk, initializing [ 2096.351132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2096.390650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.656227] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2102.694718] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2104.439602] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2106.379238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2106.384913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2106.446092] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2106.619965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2108.102720] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2108.218728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2111.463161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2111.463802] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2111.471826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2111.478842] Lustre: Skipped 1 previous similar message [ 2115.247117] LustreError: 140965:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2115.285980] Lustre: server umount lustre-OST0000 complete [ 2121.185585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2121.187462] LustreError: Skipped 1 previous similar message [ 2121.188537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2121.191435] Lustre: Skipped 1 previous similar message [ 2121.193144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.149803] LustreError: 141169:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2123.151419] LustreError: 141169:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2123.185680] Lustre: server umount lustre-MDT0000 complete [ 2124.691840] LustreError: 139787:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964835 with bad export cookie 14714726746817776682 [ 2124.698637] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2124.698704] LustreError: 139787:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2124.787896] LustreError: 141371:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2124.790130] LustreError: 141371:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2124.854932] Lustre: server umount lustre-MDT0001 complete [ 2128.526116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.733481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2130.312394] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2133.987391] LustreError: 141847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2133.996513] LustreError: 141847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2134.084550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2134.284092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2136.172381] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2138.131692] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2139.513564] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2142.984271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2143.168137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.806799] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2148.264259] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2149.591091] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2153.803695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2155.109854] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2155.188186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2159.193363] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2159.222150] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2159.295803] Lustre: lustre-OST0001: new disk, initializing [ 2159.297576] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2159.336645] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2162.112567] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2166.224218] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2167.344431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2167.350567] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2167.409111] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2168.421477] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2170.322471] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2170.445194] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2172.101472] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff96ff44aaf000.ost_server_uuid 50 [ 2172.810681] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff96ff44aaf000.ost_server_uuid in FULL state after 0 sec [ 2177.514121] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.515349] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2177.517763] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2177.517771] Lustre: Skipped 1 previous similar message [ 2177.522268] Lustre: Skipped 2 previous similar messages [ 2181.363174] LustreError: 146062:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2181.367214] LustreError: 146062:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2181.398678] Lustre: server umount lustre-OST0001 complete [ 2182.631541] LustreError: 143574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2182.642246] LustreError: 143574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2184.161767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2184.166214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2184.173860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2184.176872] Lustre: Skipped 1 previous similar message [ 2187.755941] LustreError: 143574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.765346] LustreError: 143574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2189.106578] LustreError: 146265:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2189.111955] LustreError: 146265:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2189.171499] Lustre: server umount lustre-OST0000 complete [ 2190.817019] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2190.819341] LustreError: Skipped 1 previous similar message [ 2190.820599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2190.824007] Lustre: Skipped 1 previous similar message [ 2190.825595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2190.828913] Lustre: Skipped 3 previous similar messages [ 2196.569884] Lustre: server umount lustre-MDT0000 complete [ 2197.697407] LustreError: 141826:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964908 with bad export cookie 14714726746817777550 [ 2197.700232] LustreError: 141826:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2197.700739] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2197.774494] LustreError: 146667:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2197.776813] LustreError: 146667:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2197.828231] Lustre: server umount lustre-MDT0001 complete [ 2204.174116] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2205.272871] Key type lgssc unregistered [ 2205.430546] LNet: 147648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2205.432732] LNetError: 147648:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2205.445507] LNet: Removed LNI 192.168.202.103@tcp [ 2205.808648] Key type .llcrypt unregistered [ 2205.811558] Key type ._llcrypt unregistered [ 2218.763789] Key type ._llcrypt registered [ 2218.764949] Key type .llcrypt registered [ 2218.797209] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 2226.219663] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2226.597296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2226.649259] alg: No test for adler32 (adler32-zlib) [ 2227.508220] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 2227.603608] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2229.192163] Key type lgssc registered [ 2229.685190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2233.303525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2235.782309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2238.296995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2240.755690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2243.799176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2243.822756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2243.835824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.935280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2244.946389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2244.982589] Lustre: lustre-MDT0000: new disk, initializing [ 2245.006497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2245.012529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2246.528556] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2251.755051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2251.788556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2251.841242] Lustre: 151554:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2251.874990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2251.878428] Lustre: Skipped 1 previous similar message [ 2251.921596] Lustre: lustre-MDT0001: new disk, initializing [ 2251.957765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2251.971539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2251.979506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2253.660613] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2257.369808] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2259.011987] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2262.242886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2262.283969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2262.409536] Lustre: lustre-OST0000: new disk, initializing [ 2262.411888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2262.445716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.990304] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2268.905394] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2270.537704] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2272.269853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2272.277727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2272.327299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2272.718096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2274.369899] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2274.434376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2277.348517] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2277.349236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2277.354575] LustreError: Skipped 1 previous similar message [ 2277.362611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2277.368667] Lustre: Skipped 1 previous similar message [ 2281.332218] LustreError: 154465:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2281.388379] Lustre: server umount lustre-OST0000 complete [ 2282.977435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2282.981821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2282.988459] Lustre: Skipped 1 previous similar message [ 2282.991706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2287.585221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2287.585837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.706483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.711024] Lustre: Skipped 2 previous similar messages [ 2297.312178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2297.373454] LustreError: 154666:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2297.377352] LustreError: 154666:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2297.409761] Lustre: server umount lustre-MDT0000 complete [ 2299.161334] LustreError: 151545:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965009 with bad export cookie 3340255674215177858 [ 2299.164698] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2299.165842] LustreError: 151545:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2299.232503] LustreError: 154868:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2299.236044] LustreError: 154868:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2299.282936] Lustre: server umount lustre-MDT0001 complete [ 2302.609795] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 22:43:34 (1761965014) [ 2309.236884] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 2313.124631] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2316.993765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2319.879295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2322.467616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2325.474170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2328.721644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2328.759540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2328.893337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2328.913319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2328.968329] Lustre: lustre-MDT0000: new disk, initializing [ 2329.003654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2329.016236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2330.608507] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2335.442263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2335.473704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2335.531512] Lustre: 157837:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2335.567584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2335.570646] Lustre: Skipped 1 previous similar message [ 2335.629855] Lustre: lustre-MDT0001: new disk, initializing [ 2335.686643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2335.708309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2335.714949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2337.395785] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2341.015472] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2342.360809] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2344.812764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2344.850384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2344.964313] Lustre: lustre-OST0000: new disk, initializing [ 2344.966349] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2344.994463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2346.747716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2346.750385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2346.777076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2347.204089] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2350.300341] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2352.534142] Lustre: Failing over lustre-OST0000 [ 2352.583146] LustreError: 160257:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2352.586948] LustreError: 160257:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2352.641906] Lustre: server umount lustre-OST0000 complete [ 2366.159567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2366.319081] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.327214] Lustre: Skipped 1 previous similar message [ 2366.327278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2366.343225] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2366.924619] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2368.817812] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2370.592957] Lustre: Failing over lustre-MDT0000 [ 2370.681911] LustreError: 161221:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2370.685229] LustreError: 161221:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2370.720560] Lustre: server umount lustre-MDT0000 complete [ 2371.558403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.559511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2371.561872] LustreError: 158909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.561883] LustreError: 158909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2371.566409] Lustre: Skipped 2 previous similar messages [ 2373.600192] Lustre: 149772:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761965068/real 1761965068] req@00000000d5f98a30 x1847554146888448/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761965084 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2376.673507] LustreError: 158615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2376.678297] LustreError: 158615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2378.720185] Lustre: 149770:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761965073/real 1761965073] req@00000000abbfa781 x1847554146890112/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761965089 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2381.794050] LustreError: 158615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.802919] LustreError: 158615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2384.771803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2384.834475] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2384.841648] Lustre: 157839:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.202.103@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2384.946945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2385.008220] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2386.739694] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2387.391891] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2389.997219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2390.010585] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2390.017310] Lustre: Skipped 1 previous similar message [ 2390.046845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2395.111552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2395.114885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2395.120886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2395.123876] Lustre: Skipped 1 previous similar message [ 2397.365738] LustreError: 162098:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2397.368584] LustreError: 162098:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2397.422827] Lustre: server umount lustre-OST0000 complete [ 2405.183536] Lustre: server umount lustre-MDT0000 complete [ 2405.345736] LustreError: 158908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2405.358798] LustreError: 158908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2406.957262] LustreError: 159569:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965117 with bad export cookie 3340255674215179909 [ 2406.962731] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2406.963561] LustreError: 159569:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2407.114114] Lustre: server umount lustre-MDT0001 complete [ 2411.657178] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2413.235706] Key type lgssc unregistered [ 2413.402687] LNet: 163027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2413.407330] LNetError: 163027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2413.415769] LNet: Removed LNI 192.168.202.103@tcp [ 2413.938780] Key type .llcrypt unregistered [ 2413.941156] Key type ._llcrypt unregistered [ 2437.036375] Key type ._llcrypt registered [ 2437.038175] Key type .llcrypt registered [ 2437.075712] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2437.390943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2437.404773] alg: No test for adler32 (adler32-zlib) [ 2438.323477] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 2438.450627] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2440.072198] Key type lgssc registered [ 2440.677756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2444.573400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2444.585865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2445.802003] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2447.494710] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2451.475532] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2453.916644] LustreError: 164544:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2453.923506] LustreError: 164544:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2453.928750] LustreError: 164544:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 2454.011944] LustreError: 165059:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2454.066673] Lustre: server umount lustre-MDT0000 complete [ 2460.975335] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 2464.897991] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2469.124768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2472.149136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2475.125679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2478.213515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2481.176065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2481.204716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2481.327353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2481.343478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2481.396135] Lustre: lustre-MDT0000: new disk, initializing [ 2481.428932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.441058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2483.233826] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2488.872423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2488.899781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2488.949351] Lustre: 167683:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2488.998458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2489.002066] Lustre: Skipped 1 previous similar message [ 2489.053372] Lustre: lustre-MDT0001: new disk, initializing [ 2489.108192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2489.127789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2489.133368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2490.994417] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2494.910811] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2496.194434] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2499.660836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2499.705749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2499.867540] Lustre: lustre-OST0000: new disk, initializing [ 2499.871198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2499.925160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2502.786274] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2506.662513] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2507.287508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2507.295490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2507.348738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2514.301846] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2515.260911] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 22:47:06 (1761965226) [ 2607.072826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2607.077568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2607.085261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2609.632703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2609.633364] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2614.752926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2614.756661] Lustre: Skipped 4 previous similar messages [ 2619.873122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2619.878074] Lustre: Skipped 2 previous similar messages [ 2620.384252] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2620.440437] LustreError: 171169:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2620.444249] LustreError: 171169:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2620.506617] Lustre: server umount lustre-MDT0000 complete [ 2622.285806] LustreError: 167676:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965333 with bad export cookie 8293294814114991924 [ 2622.287913] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2622.297099] LustreError: 167676:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2624.992837] LustreError: 167689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.993049] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2625.006498] LustreError: 167689:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2625.020646] Lustre: Skipped 1 previous similar message [ 2625.023932] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2628.000792] LustreError: 167689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.425324] LustreError: 171372:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2628.429040] LustreError: 171372:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2628.494163] Lustre: server umount lustre-MDT0001 complete [ 2636.523693] LustreError: 171574:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2636.525586] LustreError: 171574:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2636.554944] Lustre: server umount lustre-OST0000 complete [ 2640.260413] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 2643.531214] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2647.974656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2650.998284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2653.891521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2656.681137] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2660.023267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2660.061292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2660.206582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2660.228272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2660.288446] Lustre: lustre-MDT0000: new disk, initializing [ 2660.316394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2660.325491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2662.122457] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2667.844865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2667.882114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2667.936211] Lustre: 173893:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2667.971866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2667.976071] Lustre: Skipped 1 previous similar message [ 2668.027384] Lustre: lustre-MDT0001: new disk, initializing [ 2668.075367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2668.078563] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2669.829798] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2673.911649] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2675.463665] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2678.632788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2678.655531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2678.751583] Lustre: lustre-OST0000: new disk, initializing [ 2678.753492] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2678.782198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2678.784159] Lustre: Skipped 1 previous similar message [ 2681.078179] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2683.404612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2683.410076] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2683.448608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2685.066772] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2686.862765] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2686.962662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2688.759238] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2688.842485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2693.604654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2693.608388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2693.610123] LustreError: Skipped 1 previous similar message [ 2693.614942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2693.620668] Lustre: Skipped 2 previous similar messages [ 2704.352094] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2704.378134] LustreError: 176793:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2704.382938] LustreError: 176793:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2704.443626] Lustre: server umount lustre-OST0000 complete [ 2708.963491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2708.963742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2708.975919] Lustre: Skipped 2 previous similar messages [ 2712.313739] Lustre: server umount lustre-MDT0000 complete [ 2714.082623] LustreError: 174684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2714.094386] LustreError: 174684:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2714.196381] LustreError: 174667:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965425 with bad export cookie 8293294814115150754 [ 2714.200066] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2714.205479] LustreError: 174667:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2714.295390] LustreError: 177197:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2714.300219] LustreError: 177197:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2714.378336] Lustre: server umount lustre-MDT0001 complete [ 2718.621851] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:50:29 (1761965429) [ 2726.668563] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 2731.481250] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2736.339808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2739.603686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2742.689087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2745.845163] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2749.102075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2749.143175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2749.266927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2749.281089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2749.332621] Lustre: lustre-MDT0000: new disk, initializing [ 2749.365195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2749.374046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2751.123644] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2756.683666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2756.726143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2756.788667] Lustre: 180166:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2756.942150] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2758.726515] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2762.862865] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2764.436521] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2767.331643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2767.354246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2769.756083] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2773.521411] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2773.577400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2773.656222] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2778.594102] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2778.597989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2778.599328] LustreError: Skipped 1 previous similar message [ 2778.606776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2778.613570] Lustre: Skipped 7 previous similar messages [ 2782.323126] LustreError: 182691:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2782.327073] LustreError: 182691:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2782.385938] Lustre: server umount lustre-OST0000 complete [ 2787.809599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2790.382709] Lustre: server umount lustre-MDT0000 complete [ 2792.063108] LustreError: 180159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965502 with bad export cookie 8293294814115151636 [ 2792.069991] LustreError: 180159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2792.070767] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2792.234497] Lustre: server umount lustre-MDT0001 complete [ 2795.646648] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:51:46 (1761965506) [ 2802.726577] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 2806.706465] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2811.268105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2814.025885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2816.339323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2818.790319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2821.448483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2821.467837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2821.565577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2821.575202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2821.576759] Lustre: Skipped 3 previous similar messages [ 2821.611339] Lustre: lustre-MDT0000: new disk, initializing [ 2821.612576] Lustre: Skipped 2 previous similar messages [ 2821.635368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2821.637864] Lustre: Skipped 2 previous similar messages [ 2821.645641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2821.649992] Lustre: Skipped 2 previous similar messages [ 2822.972248] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2827.750859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2827.790276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2827.849876] Lustre: 186064:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2828.020358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2829.621849] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2833.665375] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2835.318376] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2838.641773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2838.688698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2841.575294] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2844.743110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2845.622286] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2849.765421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2849.765553] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2849.767287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2849.767296] Lustre: Skipped 3 previous similar messages [ 2849.772636] Lustre: Skipped 4 previous similar messages [ 2849.777097] LustreError: Skipped 1 previous similar message [ 2853.812109] LustreError: 188589:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2853.816045] LustreError: 188589:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2853.876710] Lustre: server umount lustre-OST0000 complete [ 2861.811896] Lustre: server umount lustre-MDT0000 complete [ 2863.009695] LustreError: 186074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2863.019052] LustreError: 186074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2863.643443] LustreError: 187805:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965574 with bad export cookie 8293294814115152721 [ 2863.646261] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2863.647143] LustreError: 187805:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2863.767436] Lustre: server umount lustre-MDT0001 complete [ 2868.669157] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2870.121287] Key type lgssc unregistered [ 2870.269531] LNet: 189516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2870.272414] LNetError: 189516:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2870.283458] LNet: Removed LNI 192.168.202.103@tcp [ 2870.766733] Key type .llcrypt unregistered [ 2870.769053] Key type ._llcrypt unregistered [ 2882.820191] Key type ._llcrypt registered [ 2882.821479] Key type .llcrypt registered [ 2882.865690] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2883.285563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2883.341990] alg: No test for adler32 (adler32-zlib) [ 2884.204843] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 2884.279176] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2885.856213] Key type lgssc registered [ 2886.452337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2889.749258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2889.755682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.933672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2892.487635] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2896.070903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2896.260647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2897.939515] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2899.830602] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2901.713865] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2905.046601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2905.223903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2907.964827] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2910.574051] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2915.503166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2931.175030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2931.177430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2931.178718] Lustre: Skipped 1 previous similar message [ 2931.181720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2934.317131] LustreError: 193128:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2934.347800] Lustre: server umount lustre-OST0000 complete [ 2936.291780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2936.300076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2936.303212] Lustre: Skipped 2 previous similar messages [ 2941.409385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2941.413511] Lustre: Skipped 1 previous similar message [ 2941.710828] LustreError: 193330:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2941.714922] LustreError: 193330:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2941.749587] Lustre: server umount lustre-MDT0000 complete [ 2943.377907] LustreError: 192487:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965654 with bad export cookie 7047571674026390271 [ 2943.382609] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2943.384583] LustreError: 192487:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2943.485796] LustreError: 193532:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2943.488541] LustreError: 193532:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2943.535795] Lustre: server umount lustre-MDT0001 complete [ 2947.537456] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2948.665628] Key type lgssc unregistered [ 2948.788458] LNet: 194056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2948.791425] LNetError: 194056:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2948.799422] LNet: Removed LNI 192.168.202.103@tcp [ 2949.126885] Key type .llcrypt unregistered [ 2949.129274] Key type ._llcrypt unregistered [ 2959.512104] Key type ._llcrypt registered [ 2959.513390] Key type .llcrypt registered [ 2959.541409] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2959.952654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2960.012164] alg: No test for adler32 (adler32-zlib) [ 2960.924677] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 2961.049639] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2962.624248] Key type lgssc registered [ 2963.168650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2966.489560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2966.501233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2967.692894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2969.243018] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2972.766335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2972.913201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2974.415315] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2976.220495] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2977.806894] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2980.228197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2980.346932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2982.505525] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2984.885092] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2992.620704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3000.777336] Lustre: lustre-OST0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 3002.856762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3002.859298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3002.860325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3002.873549] Lustre: Skipped 1 previous similar message [ 3004.405173] LustreError: 197667:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3004.471562] Lustre: server umount lustre-OST0000 complete [ 3007.128401] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff96ff504fa000.ost_server_uuid 50 [ 3007.988360] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3019.233671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3019.238310] LustreError: Skipped 1 previous similar message [ 3019.240556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3019.247250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3019.249788] Lustre: Skipped 1 previous similar message [ 3022.309057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3022.309083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3024.471766] LustreError: 198016:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3024.476082] LustreError: 198016:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3024.527621] Lustre: server umount lustre-MDT0000 complete [ 3025.913578] LustreError: 195401:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965736 with bad export cookie 8644850699794011877 [ 3025.917964] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3025.983256] LustreError: 198219:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3025.985017] LustreError: 198219:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3026.036922] Lustre: server umount lustre-MDT0001 complete [ 3028.652203] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 22:55:40 (1761965740) [ 3030.727471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3030.867812] LustreError: 198888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3030.872769] LustreError: 198888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3030.893319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3032.080156] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3034.931155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3035.068382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3036.825703] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3039.363932] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3041.113353] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3044.392359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3044.543942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.051533] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3049.686139] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3053.096096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3053.148373] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3053.257491] Lustre: lustre-OST0001: new disk, initializing [ 3053.262095] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3055.982321] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3056.438834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3059.223005] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3059.229121] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3059.273548] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3059.938822] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3061.729366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3061.733879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3061.742166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3061.745763] Lustre: Skipped 1 previous similar message [ 3065.321651] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3067.316492] LustreError: 202067:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3067.320421] LustreError: 202067:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3067.347288] Lustre: server umount lustre-OST0000 complete [ 3070.081566] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3070.186294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3070.401293] LustreError: 200469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3071.961105] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3072.029862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3075.520979] LustreError: 200470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3075.531023] LustreError: 200470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3080.642181] LustreError: 200469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3080.652475] LustreError: 200469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3084.770305] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3084.770387] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3084.775032] LustreError: Skipped 1 previous similar message [ 3084.775947] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3084.782545] Lustre: Skipped 2 previous similar messages [ 3084.791264] Lustre: Skipped 4 previous similar messages [ 3085.798169] LustreError: 200469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3085.808683] LustreError: 200469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3088.499153] LustreError: 202748:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3088.503170] LustreError: 202748:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3088.558665] Lustre: server umount lustre-OST0001 complete [ 3096.532185] LustreError: 202949:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3096.535949] LustreError: 202949:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3096.570079] Lustre: server umount lustre-MDT0000 complete [ 3098.388205] LustreError: 200491:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965809 with bad export cookie 8644850699794012941 [ 3098.392775] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3098.398496] LustreError: 200491:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3098.544599] Lustre: server umount lustre-MDT0001 complete [ 3101.261726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3103.090074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3104.949451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3106.957176] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3110.157911] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 22:57:01 (1761965821) [ 3113.503213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3113.688267] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3113.708094] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3113.837940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3113.842137] Lustre: Skipped 1 previous similar message [ 3115.764931] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3120.023737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3120.078918] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3120.102772] Lustre: 204641:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3122.146944] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3124.831981] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3126.679136] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3129.976861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3130.105735] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3132.845047] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3135.210563] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3136.077107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 3138.610588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3138.638588] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3141.239131] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3142.601173] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3143.730575] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3146.211079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3146.211569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3146.215663] LustreError: Skipped 1 previous similar message [ 3146.222994] Lustre: Skipped 2 previous similar messages [ 3146.223832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3146.232030] Lustre: Skipped 7 previous similar messages [ 3151.155198] LustreError: 207811:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3151.159217] LustreError: 207811:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3151.188743] Lustre: server umount lustre-OST0000 complete [ 3152.868445] LustreError: 206255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3152.877770] LustreError: 206255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3163.108634] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3163.109080] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3163.110989] LustreError: Skipped 1 previous similar message [ 3163.114579] Lustre: Skipped 1 previous similar message [ 3168.551890] Lustre: server umount lustre-OST0001 complete [ 3176.418674] Lustre: server umount lustre-MDT0000 complete [ 3177.941144] LustreError: 204632:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965888 with bad export cookie 8644850699794014327 [ 3177.943854] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3177.944298] LustreError: 204632:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3178.057669] Lustre: server umount lustre-MDT0001 complete [ 3180.658706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3182.620691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3184.456921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3186.221613] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3189.284752] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 22:58:20 (1761965900) [ 3196.397175] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 3200.216233] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3204.186794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3207.213606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3210.159270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3213.408893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3216.847485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3216.889270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3217.028852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3217.045145] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3217.095793] Lustre: lustre-MDT0000: new disk, initializing [ 3217.126487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3217.129974] Lustre: Skipped 3 previous similar messages [ 3217.139141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3218.889046] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3224.651817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3224.690531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3224.742546] Lustre: 212218:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3224.776757] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3224.780442] Lustre: Skipped 1 previous similar message [ 3224.828681] Lustre: lustre-MDT0001: new disk, initializing [ 3224.895741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3224.901124] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3226.533516] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3230.581519] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3232.314738] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3235.648100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3235.684225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3235.829671] Lustre: lustre-OST0000: new disk, initializing [ 3235.832338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3238.343302] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3240.463737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3240.470214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3240.526762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3242.391768] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3244.039019] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3244.119119] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3245.838382] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3245.950337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3250.665453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3250.666172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3250.670120] LustreError: Skipped 2 previous similar messages [ 3250.677611] Lustre: Skipped 3 previous similar messages [ 3250.681145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3250.686466] Lustre: Skipped 11 previous similar messages [ 3261.408150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3261.436137] LustreError: 215115:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3261.440042] LustreError: 215115:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3261.504881] Lustre: server umount lustre-OST0000 complete [ 3269.488922] Lustre: server umount lustre-MDT0000 complete [ 3271.096579] LustreError: 212208:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965981 with bad export cookie 8644850699794016098 [ 3271.103236] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3271.103555] LustreError: 212208:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3271.270638] Lustre: server umount lustre-MDT0001 complete [ 3275.219264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3275.457629] LustreError: 215995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3275.468866] LustreError: 215995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3277.122144] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3281.154086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3281.362303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3281.368527] Lustre: Skipped 3 previous similar messages [ 3283.188257] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3285.568266] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3287.269984] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3290.518623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3293.209536] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3295.797262] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3297.682197] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3301.936896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3303.836143] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3303.950392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3311.024514] Lustre: server umount lustre-OST0000 complete [ 3313.738837] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3315.617919] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3340.431646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3343.260719] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3345.797610] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3346.568500] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3346.796393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3346.803389] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3346.803495] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3352.038091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3352.042877] LustreError: Skipped 2 previous similar messages [ 3352.045587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3352.045596] Lustre: Skipped 4 previous similar messages [ 3352.046559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3352.059028] Lustre: Skipped 9 previous similar messages [ 3353.967994] LustreError: 220342:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3353.972218] LustreError: 220342:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3354.029414] Lustre: server umount lustre-OST0000 complete [ 3362.004885] Lustre: server umount lustre-MDT0000 complete [ 3362.274675] LustreError: 215995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3362.283735] LustreError: 215995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3363.821592] LustreError: 216706:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761966074 with bad export cookie 8644850699794016980 [ 3363.831568] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3364.005411] Lustre: server umount lustre-MDT0001 complete [ 3368.162526] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:01:19 (1761966079) [ 3371.364823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3373.271580] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3377.104765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3378.742701] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3381.204579] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3382.931077] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3386.251509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3389.018572] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3391.593567] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3393.537511] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3397.814428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3399.678324] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3399.788563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3403.043292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3403.084448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3403.171330] Lustre: lustre-OST0001: new disk, initializing [ 3403.173634] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3405.929415] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3409.943260] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3409.946111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3409.954558] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3410.003255] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3411.796181] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3411.908856] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3413.744331] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3413.853766] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3416.851333] Lustre: server umount lustre-OST0001 complete [ 3419.533269] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3419.661494] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3421.532331] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3421.649571] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3427.266098] LustreError: 223011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3427.277118] LustreError: 223011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3446.263586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3446.392662] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3446.395745] Lustre: Skipped 6 previous similar messages [ 3448.602544] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3451.034426] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3451.848943] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3451.895460] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3451.902618] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3451.902866] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3451.906197] Lustre: Skipped 1 previous similar message [ 3467.232152] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3467.287789] Lustre: server umount lustre-OST0001 complete [ 3482.597869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3482.598579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3482.601451] Lustre: Skipped 14 previous similar messages [ 3482.608610] Lustre: Skipped 8 previous similar messages [ 3494.368187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3494.395178] LustreError: 227434:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3494.399130] LustreError: 227434:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3494.455938] Lustre: server umount lustre-OST0000 complete [ 3502.447793] Lustre: server umount lustre-MDT0000 complete [ 3504.249465] LustreError: 221407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761966215 with bad export cookie 8644850699794017939 [ 3504.256568] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3504.259093] LustreError: 221407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3510.479649] Lustre: server umount lustre-MDT0001 complete [ 3513.036240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3514.742956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3516.511744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3518.363862] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3521.532596] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:03:52 (1761966232) [ 3524.777888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3524.934606] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3524.954069] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3526.864519] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3530.994603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3531.060812] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3531.078829] Lustre: 229336:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3532.989217] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3535.680172] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3537.466868] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3540.590761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3540.720322] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3543.136354] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3545.630405] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3555.524767] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3555.574887] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3558.445652] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3561.000355] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3561.787988] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3563.567761] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3563.679881] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3565.557803] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3565.677917] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3567.312899] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff96ff44aa8800.ost_server_uuid 50 [ 3567.980844] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff96ff44aa8800.ost_server_uuid in FULL state after 0 sec [ 3568.427133] Lustre: Permanently deactivating lustre-OST0001 [ 3568.432163] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3569.426252] Lustre: Permanently reactivating lustre-OST0001 [ 3573.227109] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3573.231691] LustreError: Skipped 6 previous similar messages [ 3576.908137] Lustre: server umount lustre-OST0001 complete [ 3578.347117] LustreError: 231508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3578.356625] LustreError: 231508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 3584.812873] Lustre: server umount lustre-OST0000 complete [ 3592.827689] Lustre: server umount lustre-MDT0000 complete [ 3594.597928] LustreError: 229327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761966305 with bad export cookie 8644850699794019213 [ 3594.607893] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3594.728821] Lustre: server umount lustre-MDT0001 complete [ 3597.355939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3599.142589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3600.796144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3602.689784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3605.474153] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:05:16 (1761966316) [ 3605.966605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3609.153715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3609.320722] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3609.338675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3609.346202] Lustre: Skipped 5 previous similar messages [ 3611.294163] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3615.267996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3615.326396] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3615.343922] Lustre: 235431:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3617.209667] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3619.551983] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3621.055769] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3624.300773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3624.337767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3624.467135] Lustre: Permanently deactivating lustre-OST0000 [ 3624.476086] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3627.212148] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3629.684547] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3630.594289] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3632.664692] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3635.462965] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3638.167762] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3638.880494] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3644.507452] Lustre: Permanently reactivating lustre-OST0000 [ 3644.517536] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 3644.522219] Lustre: Skipped 2 previous similar messages [ 3646.743351] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3646.748116] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3646.757437] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3646.760957] Lustre: Skipped 1 previous similar message [ 3646.819658] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3646.834976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3656.004324] Lustre: server umount lustre-OST0001 complete [ 3663.861531] Lustre: server umount lustre-OST0000 complete [ 3672.009969] Lustre: server umount lustre-MDT0000 complete [ 3673.934355] LustreError: 235422:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761966384 with bad export cookie 8644850699794020865 [ 3673.944698] LustreError: 235422:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3673.945539] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3674.107469] Lustre: server umount lustre-MDT0001 complete [ 3679.202890] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3680.676478] Key type lgssc unregistered [ 3680.839557] LNet: 239792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3680.844362] LNetError: 239792:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3680.854483] LNet: Removed LNI 192.168.202.103@tcp [ 3681.267177] Key type .llcrypt unregistered [ 3681.269173] Key type ._llcrypt unregistered [ 3692.637843] Key type ._llcrypt registered [ 3692.640406] Key type .llcrypt registered [ 3692.686403] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3693.127902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3693.179139] alg: No test for adler32 (adler32-zlib) [ 3694.110415] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 3694.235057] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3695.856226] Key type lgssc registered [ 3696.453480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3698.098657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3701.402293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3701.413302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3702.699486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3704.672280] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3708.993809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3709.036510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3709.085330] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3709.098661] Lustre: 241199:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3709.103069] Lustre: Permanently deactivating lustre-MDT0001 [ 3709.108954] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3709.261673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3711.013410] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3712.400210] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3713.537025] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3715.213689] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3718.540226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3718.751954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3719.784130] LustreError: 242948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3719.789971] LustreError: 242948:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3719.791106] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3721.444219] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3723.985657] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3725.286379] LustreError: 242812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3727.280795] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3727.335450] LustreError: 242812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3727.344414] LustreError: 242812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3727.354624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3727.456229] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3728.194898] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3728.215443] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3729.785487] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3731.935350] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3733.629482] Lustre: Permanently reactivating lustre-MDT0001 [ 3733.639342] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3733.643228] Lustre: Skipped 2 previous similar messages [ 3734.875935] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3734.884518] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3734.889962] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3734.898094] LustreError: 244306:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 3734.898965] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3734.903981] LustreError: 244306:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3734.913489] LustreError: 244306:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.103@tcp: cfg command failed: rc = -108 [ 3734.918727] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 3734.918727] [ 3734.923206] LustreError: 241201:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3737.419819] Lustre: lustre-MDT0001: Client 50c379f7-7ee4-4d0b-95a9-e370cb495769 (at 192.168.202.3@tcp) reconnecting [ 3739.541453] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3740.353121] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3742.606193] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3742.686739] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3744.095686] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3744.169781] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3744.540098] Lustre: 244241:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 3744.543291] Lustre: 244241:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 3744.545423] Lustre: 244241:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 3744.547629] Lustre: 244241:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 3744.549906] Lustre: 244241:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 3744.552008] Lustre: 244241:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 3745.003409] Lustre: Permanently deactivating lustre-MDT0001 [ 3745.009079] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3745.010717] Lustre: Skipped 2 previous similar messages [ 3753.010543] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3756.251731] Lustre: 241204:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 504 < left 979, rollback = 2 [ 3756.256667] Lustre: 241204:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 1 previous similar message [ 3756.260382] Lustre: 241204:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/8, destroy: 0/0/0 [ 3756.263995] Lustre: 241204:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3756.267658] Lustre: 241204:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 6/979/0 [ 3756.271082] Lustre: 241204:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3756.274206] Lustre: 241204:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3756.277419] Lustre: 241204:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3756.279872] Lustre: 241204:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 8/130/4, delete: 0/0/0 [ 3756.282527] Lustre: 241204:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3756.285098] Lustre: 241204:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0 [ 3756.287845] Lustre: 241204:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3758.056384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3758.060558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3758.065110] Lustre: Skipped 3 previous similar messages [ 3758.068697] Lustre: Skipped 2 previous similar messages [ 3763.171553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3763.175263] Lustre: Skipped 2 previous similar messages [ 3763.797880] LustreError: 245246:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3763.846552] Lustre: server umount lustre-MDT0000 complete [ 3765.504613] LustreError: 241927:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761966476 with bad export cookie 1922706340908204559 [ 3765.510251] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3765.516143] LustreError: 241927:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3765.610995] LustreError: 245447:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3765.613343] LustreError: 245447:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3765.672215] Lustre: server umount lustre-MDT0001 complete [ 3773.556248] LustreError: 245647:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3773.560207] LustreError: 245647:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3773.584128] Lustre: server umount lustre-OST0000 complete [ 3778.758086] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:08:10 (1761966490) [ 3783.851413] LustreError: 246653:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3783.855359] LustreError: 246653:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3783.924514] Lustre: server umount lustre-OST0001 complete [ 3786.829932] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 3791.183323] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3795.796318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3798.858572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3801.561575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3804.334366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3807.250114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3807.286077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3807.423275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3807.429452] Lustre: Skipped 2 previous similar messages [ 3807.445256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3807.495058] Lustre: lustre-MDT0000: new disk, initializing [ 3807.529846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3807.540771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3809.163605] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3814.384535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3814.416905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3814.454557] Lustre: 248819:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3814.496880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3814.500365] Lustre: Skipped 1 previous similar message [ 3814.550375] Lustre: lustre-MDT0001: new disk, initializing [ 3814.614104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3814.619810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3816.100098] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3820.008680] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3821.675360] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3824.971626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3825.010969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3825.162802] Lustre: lustre-OST0000: new disk, initializing [ 3825.166484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3825.212320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3825.215659] Lustre: Skipped 1 previous similar message [ 3827.667561] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3831.590367] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3834.379521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3834.385568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3834.440652] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3837.645676] LustreError: 248826:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3842.709206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3842.747149] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3842.842877] Lustre: lustre-OST0001: new disk, initializing [ 3842.846598] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3842.897911] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3845.389787] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3847.736087] LustreError: 248826:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3847.745477] LustreError: 248826:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3849.126697] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3849.931040] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3850.274609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3850.279971] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3850.336728] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3857.848125] LustreError: 248826:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3860.458604] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3860.460119] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.472176] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3860.475588] Lustre: Skipped 2 previous similar messages [ 3862.477570] Lustre: lustre-OST0001: Not available for connect from 192.168.202.3@tcp (stopping) [ 3867.585942] Lustre: lustre-OST0001: Not available for connect from 192.168.202.3@tcp (stopping) [ 3867.590696] Lustre: Skipped 2 previous similar messages [ 3873.248182] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3873.270139] LustreError: 252284:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3873.272838] LustreError: 252284:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3873.299771] Lustre: server umount lustre-OST0001 complete [ 3875.812854] LustreError: 250536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3875.818884] LustreError: 250536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3880.938661] LustreError: 250536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3880.948357] LustreError: 250536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3886.056367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3886.059139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3886.062169] Lustre: Skipped 2 previous similar messages [ 3886.065271] Lustre: Skipped 3 previous similar messages [ 3891.180345] LustreError: 250534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3891.189841] LustreError: 250534:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3900.384125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3900.411163] LustreError: 252509:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3900.415110] LustreError: 252509:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3900.480192] Lustre: server umount lustre-OST0000 complete [ 3906.531912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3906.533383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3906.538085] Lustre: Skipped 1 previous similar message [ 3906.544026] Lustre: Skipped 5 previous similar messages [ 3908.544434] Lustre: server umount lustre-MDT0000 complete [ 3910.377598] LustreError: 249592:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761966621 with bad export cookie 1922706340908207373 [ 3910.383939] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3910.384578] LustreError: 249592:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3916.560358] LustreError: 252913:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3916.564532] LustreError: 252913:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3916.655711] Lustre: server umount lustre-MDT0001 complete [ 3921.174515] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3922.733342] Key type lgssc unregistered [ 3922.914534] LNet: 253438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3922.918888] LNetError: 253438:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3922.927734] LNet: Removed LNI 192.168.202.103@tcp [ 3923.391939] Key type .llcrypt unregistered [ 3923.393968] Key type ._llcrypt unregistered [ 3938.859119] Key type ._llcrypt registered [ 3938.861161] Key type .llcrypt registered [ 3938.908851] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 3946.518445] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3947.009588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3947.082825] alg: No test for adler32 (adler32-zlib) [ 3948.000760] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 3948.124239] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3949.736187] Key type lgssc registered [ 3950.371230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3954.630152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3957.592508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3960.319511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3963.412718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3966.652405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3966.683639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3966.692951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3967.831721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3967.849491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3967.897740] Lustre: lustre-MDT0000: new disk, initializing [ 3967.935343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3967.945769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3969.812239] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3975.434209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3975.473458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3975.514805] Lustre: 257347:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3975.541287] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3975.542887] Lustre: Skipped 1 previous similar message [ 3975.589050] Lustre: lustre-MDT0001: new disk, initializing [ 3975.629525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3975.645658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3975.650772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3977.379567] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3981.471285] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3983.235537] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3986.636712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3986.672511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3986.835111] Lustre: lustre-OST0000: new disk, initializing [ 3986.838872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3986.895448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3989.424148] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3993.104115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3993.111233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3993.173650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3993.493983] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3995.537350] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3995.660333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3997.498306] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3997.612149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4003.301279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4003.303412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4003.304431] Lustre: Skipped 1 previous similar message [ 4003.304445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4003.308702] LustreError: Skipped 1 previous similar message [ 4003.309994] Lustre: Skipped 1 previous similar message [ 4004.661196] LustreError: 260240:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4004.725969] Lustre: server umount lustre-OST0000 complete [ 4008.418234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4008.421923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4008.423315] Lustre: Skipped 1 previous similar message [ 4012.756945] LustreError: 260443:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4012.760875] LustreError: 260443:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4012.799106] Lustre: server umount lustre-MDT0000 complete [ 4014.613597] LustreError: 257339:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761966725 with bad export cookie 16542168377822408801 [ 4014.621988] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4014.714229] LustreError: 260645:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4014.717954] LustreError: 260645:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4014.805074] Lustre: server umount lustre-MDT0001 complete [ 4018.795327] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 23:12:10 (1761966730) [ 4021.982106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4022.218793] LustreError: 261316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4022.227175] LustreError: 261316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4022.261185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4024.054536] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4027.363992] LustreError: 261316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4027.897660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4028.070847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4029.890021] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4032.494799] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4034.231100] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4037.499095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4037.667596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4040.224564] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4042.809618] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4093.926728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4093.928976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4093.930909] LustreError: Skipped 1 previous similar message [ 4093.937751] Lustre: Skipped 1 previous similar message [ 4093.939401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4093.947059] Lustre: Skipped 1 previous similar message [ 4098.038203] LustreError: 263616:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4098.042123] LustreError: 263616:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4098.127465] Lustre: server umount lustre-OST0000 complete [ 4100.285662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4106.414885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4106.618479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4109.335220] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4111.931890] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4113.773858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4113.780664] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4113.780922] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4113.786590] Lustre: Skipped 1 previous similar message [ 4149.736476] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4149.741310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4149.745296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4149.749533] Lustre: Skipped 2 previous similar messages [ 4152.179157] LustreError: 265276:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4152.182686] LustreError: 265276:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4152.263707] Lustre: server umount lustre-OST0000 complete [ 4154.851602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4154.853057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4154.859351] Lustre: Skipped 1 previous similar message [ 4154.864392] Lustre: Skipped 1 previous similar message [ 4160.275199] LustreError: 265478:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4160.278923] LustreError: 265478:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4160.318335] Lustre: server umount lustre-MDT0000 complete [ 4162.136072] LustreError: 261341:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761966872 with bad export cookie 16542168377822409669 [ 4162.142885] LustreError: 261341:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4162.145727] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4162.311940] Lustre: server umount lustre-MDT0001 complete [ 4167.256267] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 4168.788439] Key type lgssc unregistered [ 4168.964755] LNet: 266204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4168.969398] LNetError: 266204:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4168.979725] LNet: Removed LNI 192.168.202.103@tcp [ 4169.449923] Key type .llcrypt unregistered [ 4169.451825] Key type ._llcrypt unregistered [ 4181.116822] Key type ._llcrypt registered [ 4181.118632] Key type .llcrypt registered [ 4181.150988] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4181.452672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4181.511755] alg: No test for adler32 (adler32-zlib) [ 4182.404315] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 4182.501751] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 4184.080189] Key type lgssc registered [ 4184.617343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4188.237524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4188.249839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4189.492459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4191.387856] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4195.587793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4195.786186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4197.656424] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4200.314927] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4202.050936] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4205.446584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4205.648452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4208.373614] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4210.952829] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4214.898688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4227.553295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4227.556936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4227.563852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4230.631469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4230.637975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4230.638944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4230.641511] Lustre: Skipped 1 previous similar message [ 4231.923185] LustreError: 270565:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4231.989263] Lustre: server umount lustre-OST0000 complete [ 4235.748239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.750685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4235.755604] Lustre: Skipped 1 previous similar message [ 4239.952727] LustreError: 270767:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4239.956506] LustreError: 270767:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4239.996383] Lustre: server umount lustre-MDT0000 complete [ 4240.867574] LustreError: 267568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4240.876908] LustreError: 267568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4241.717474] LustreError: 270768:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761966952 with bad export cookie 1380695600558056808 [ 4241.724567] LustreError: 270768:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4241.725799] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4241.825847] LustreError: 270970:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4241.829200] LustreError: 270970:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4241.907561] Lustre: server umount lustre-MDT0001 complete [ 4246.352479] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 4247.896825] Key type lgssc unregistered [ 4248.086633] LNet: 271494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4248.091281] LNetError: 271494:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4248.100776] LNet: Removed LNI 192.168.202.103@tcp [ 4248.556375] Key type .llcrypt unregistered [ 4248.559826] Key type ._llcrypt unregistered [ 4257.539856] Key type ._llcrypt registered [ 4257.541390] Key type .llcrypt registered [ 4257.584789] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4258.038879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4258.077818] alg: No test for adler32 (adler32-zlib) [ 4258.993193] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 4259.118395] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 4260.712209] Key type lgssc registered [ 4261.314287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4264.871651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4264.882972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4266.107336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4267.839941] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4271.823957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4272.022225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4273.782058] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4276.383250] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4277.977518] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4281.197810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4281.253472] LustreError: 274250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4281.260515] LustreError: 274250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4281.378715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4283.890130] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4286.351189] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4290.676177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4298.209255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4298.213986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4298.222071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4301.285846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4301.287628] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4301.287872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4301.289636] Lustre: Skipped 1 previous similar message [ 4302.323169] LustreError: 275037:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4302.377459] Lustre: server umount lustre-OST0000 complete [ 4306.403187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4306.404442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4306.410659] Lustre: Skipped 1 previous similar message [ 4306.416797] Lustre: Skipped 1 previous similar message [ 4310.285459] LustreError: 275239:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4310.289216] LustreError: 275239:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4310.322505] Lustre: server umount lustre-MDT0000 complete [ 4311.521708] LustreError: 272665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4311.531166] LustreError: 272665:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4312.042077] LustreError: 272645:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761967022 with bad export cookie 8420847819157033227 [ 4312.047626] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4312.048614] LustreError: 272645:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4312.148626] LustreError: 275441:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4312.151364] LustreError: 275441:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4312.202953] Lustre: server umount lustre-MDT0001 complete [ 4316.670866] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 4318.015743] Key type lgssc unregistered [ 4318.191636] LNet: 275965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4318.195791] LNetError: 275965:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4318.204602] LNet: Removed LNI 192.168.202.103@tcp [ 4318.676538] Key type .llcrypt unregistered [ 4318.678688] Key type ._llcrypt unregistered [ 4329.273025] Key type ._llcrypt registered [ 4329.274547] Key type .llcrypt registered [ 4329.308131] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4329.731204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4329.744441] alg: No test for adler32 (adler32-zlib) [ 4330.630806] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 4330.732416] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 4332.312177] Key type lgssc registered [ 4332.937818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4336.712496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4336.723973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4337.952504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4339.413956] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4343.286169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4343.481097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4345.046225] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4347.532345] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4349.171845] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4352.420920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4352.600838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4355.069924] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4357.593299] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4362.863504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4373.480132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4373.480292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4373.481750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4373.495859] Lustre: Skipped 1 previous similar message [ 4378.596139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4378.599780] Lustre: Skipped 2 previous similar messages [ 4378.866153] LustreError: 280344:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4378.921892] Lustre: server umount lustre-OST0000 complete [ 4383.715320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.724064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4383.727813] Lustre: Skipped 1 previous similar message [ 4386.830282] LustreError: 280546:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4386.833938] LustreError: 280546:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4386.867844] Lustre: server umount lustre-MDT0000 complete [ 4388.560348] LustreError: 278953:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761967099 with bad export cookie 15809997412548448922 [ 4388.565647] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4388.567336] LustreError: 278953:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4388.664390] LustreError: 280749:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4388.667871] LustreError: 280749:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4388.724428] Lustre: server umount lustre-MDT0001 complete [ 4393.082542] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 4394.485230] Key type lgssc unregistered [ 4394.657671] LNet: 281273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4394.662398] LNetError: 281273:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4394.675573] LNet: Removed LNI 192.168.202.103@tcp [ 4395.114569] Key type .llcrypt unregistered [ 4395.116682] Key type ._llcrypt unregistered [ 4403.441683] Key type ._llcrypt registered [ 4403.443295] Key type .llcrypt registered [ 4403.472367] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4403.769571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4403.817870] alg: No test for adler32 (adler32-zlib) [ 4404.662282] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 4404.750207] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 4406.328169] Key type lgssc registered [ 4406.735216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4410.467151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4410.479226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4411.695035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4413.371654] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4417.577617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4417.779915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4419.555570] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4422.124984] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4423.681158] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4427.011368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4427.191123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4429.906378] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4432.455352] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4436.404286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4443.619256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4443.623895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4443.632190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4446.696582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4446.698230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4446.711304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4448.372216] LustreError: 284851:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4448.437374] Lustre: server umount lustre-OST0000 complete [ 4451.811954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4451.820837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4451.824441] Lustre: Skipped 2 previous similar messages [ 4456.464798] LustreError: 285053:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4456.468524] LustreError: 285053:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4456.508519] Lustre: server umount lustre-MDT0000 complete [ 4456.931485] LustreError: 282472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4456.941062] LustreError: 282472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4458.321973] LustreError: 283197:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761967169 with bad export cookie 9650653829685998198 [ 4458.327726] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4458.328726] LustreError: 283197:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4458.424182] LustreError: 285257:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4458.427542] LustreError: 285257:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4458.496940] Lustre: server umount lustre-MDT0001 complete [ 4463.150421] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 4464.441954] Key type lgssc unregistered [ 4464.603588] LNet: 285782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4464.608320] LNetError: 285782:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4464.617671] LNet: Removed LNI 192.168.202.103@tcp [ 4465.075907] Key type .llcrypt unregistered [ 4465.078046] Key type ._llcrypt unregistered [ 4469.937747] Key type ._llcrypt registered [ 4469.939758] Key type .llcrypt registered [ 4469.993840] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4478.007713] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 4485.650447] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4485.930615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4485.991222] alg: No test for adler32 (adler32-zlib) [ 4486.834850] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 4486.911700] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 4488.488211] Key type lgssc registered [ 4489.057299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4492.975977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4495.890712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4498.704501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4501.681352] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4504.851606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4504.881709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4504.890536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4505.999176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4506.016186] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4506.070269] Lustre: lustre-MDT0000: new disk, initializing [ 4506.107867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4506.120347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4507.816232] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4513.306331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4513.344647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4513.397587] Lustre: 289242:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4513.438661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4513.442027] Lustre: Skipped 1 previous similar message [ 4513.492619] Lustre: lustre-MDT0001: new disk, initializing [ 4513.547204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4513.567517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4513.573619] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4515.306895] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4519.224447] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4520.857314] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4523.894049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4523.933072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4524.099224] Lustre: lustre-OST0000: new disk, initializing [ 4524.102865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4524.161061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4526.709406] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4529.675501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4529.681753] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4529.732352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4530.385914] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4531.852603] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4531.960816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4533.687054] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4533.774861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4534.756955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4534.757939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4534.770604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4534.774376] Lustre: Skipped 1 previous similar message [ 4536.628175] LustreError: 292143:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4536.688621] Lustre: server umount lustre-OST0000 complete [ 4539.361582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4539.368244] LustreError: Skipped 1 previous similar message [ 4539.372190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4539.382896] Lustre: Skipped 1 previous similar message [ 4539.387440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4544.592085] LustreError: 292344:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4544.596117] LustreError: 292344:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4544.636693] Lustre: server umount lustre-MDT0000 complete [ 4546.051272] LustreError: 292346:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761967256 with bad export cookie 5061892570739955401 [ 4546.054553] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4546.054901] LustreError: 292346:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4546.118066] LustreError: 292548:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4546.120891] LustreError: 292548:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4546.176939] Lustre: server umount lustre-MDT0001 complete [ 4549.732634] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:21:01 (1761967261) [ 4552.828963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4553.027461] LustreError: 293218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4553.032267] LustreError: 293218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4553.047738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4554.262086] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4558.074605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4558.241153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4559.977819] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4562.484907] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4564.167188] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4567.222055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4567.389930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4569.729961] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4572.174494] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4582.884915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.887391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4582.889447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4582.889454] Lustre: Skipped 1 previous similar message [ 4582.891851] Lustre: Skipped 2 previous similar messages [ 4587.058127] LustreError: 295444:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4587.061969] LustreError: 295444:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4587.119248] Lustre: server umount lustre-OST0000 complete [ 4589.025198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4589.029216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4589.035692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4589.038817] Lustre: Skipped 1 previous similar message [ 4593.057849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4593.061509] Lustre: Skipped 3 previous similar messages [ 4602.338414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4602.341943] Lustre: Skipped 3 previous similar messages [ 4603.360247] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4603.417321] LustreError: 295646:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4603.421022] LustreError: 295646:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4603.459488] Lustre: server umount lustre-MDT0000 complete [ 4605.152859] LustreError: 293199:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761967315 with bad export cookie 5061892570739956269 [ 4605.159203] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4605.159732] LustreError: 293199:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4605.299778] Lustre: server umount lustre-MDT0001 complete [ 4609.764760] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 4611.234257] Key type lgssc unregistered [ 4611.385491] LNet: 296374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4611.390302] LNetError: 296374:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4611.400302] LNet: Removed LNI 192.168.202.103@tcp [ 4611.840479] Key type .llcrypt unregistered [ 4611.842608] Key type ._llcrypt unregistered [ 4623.088166] Key type ._llcrypt registered [ 4623.090021] Key type .llcrypt registered [ 4623.135168] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 4630.980493] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4631.319381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4631.359907] alg: No test for adler32 (adler32-zlib) [ 4632.222122] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 4632.321619] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 4633.904151] Key type lgssc registered [ 4634.442609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4638.578329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4641.689289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4644.604611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4647.669818] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4650.100571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4652.565644] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4655.779650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4655.812677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4655.821128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4656.920889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4656.934432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4656.972947] Lustre: lustre-MDT0000: new disk, initializing [ 4657.000606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4657.008104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4658.720596] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4664.148693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4664.183238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4664.237868] Lustre: 300138:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4664.278129] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4664.281556] Lustre: Skipped 1 previous similar message [ 4664.325415] Lustre: lustre-MDT0001: new disk, initializing [ 4664.359254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4664.376818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4664.382155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4665.996526] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4669.960618] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4671.614637] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4674.819746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4674.859966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4675.008767] Lustre: lustre-OST2710: new disk, initializing [ 4675.012479] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4675.061339] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4677.567909] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4681.441926] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4683.281568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4683.287816] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4683.338181] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4684.681360] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4684.720818] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4684.806805] Lustre: lustre-OST03e8: new disk, initializing [ 4684.810378] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4684.858657] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4687.337528] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4689.433581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4689.439794] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4689.489984] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4689.826536] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4692.989266] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4693.097259] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4694.803207] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4694.902507] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4696.658739] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4696.753377] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4698.376238] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4698.479338] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4700.129500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4700.135094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4700.143247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4704.740124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4704.745250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4704.752526] Lustre: Skipped 2 previous similar messages [ 4705.873346] LustreError: 304407:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4705.914311] Lustre: server umount lustre-MDT0000 complete [ 4707.616891] LustreError: 300128:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761967418 with bad export cookie 16615298979534985422 [ 4707.623664] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4707.623745] LustreError: 300128:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4707.719263] LustreError: 304610:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4707.722070] LustreError: 304610:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4707.776447] Lustre: server umount lustre-MDT0001 complete [ 4720.058134] LustreError: 304811:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4720.061915] LustreError: 304811:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4720.089328] Lustre: server umount lustre-OST2710 complete [ 4732.088965] LustreError: 305012:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4732.092715] LustreError: 305012:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4732.160834] Lustre: server umount lustre-OST03e8 complete [ 4740.366371] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 4744.441968] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4748.361242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4751.403249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4754.287258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4757.309669] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4760.414919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4760.449083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4760.566065] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4760.581801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4760.622704] Lustre: lustre-MDT0000: new disk, initializing [ 4760.651633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4760.658911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4762.258399] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4767.709701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4767.750934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4767.805197] Lustre: 307791:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4767.950708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4769.584318] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4773.470246] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4775.211201] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4778.593175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4778.638963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4778.788448] Lustre: lustre-OST0000: new disk, initializing [ 4778.792177] Lustre: Skipped 1 previous similar message [ 4778.796664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4778.801681] Lustre: Skipped 2 previous similar messages [ 4778.852975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4778.857235] Lustre: Skipped 1 previous similar message [ 4781.544721] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4785.535784] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4787.205629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4787.208529] Lustre: Skipped 1 previous similar message [ 4787.210709] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4787.251573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4787.370868] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4787.475580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4789.367359] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4789.483042] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4792.296540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4792.298280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4792.300421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4792.300438] Lustre: Skipped 3 previous similar messages [ 4792.301279] LustreError: Skipped 1 previous similar message [ 4792.316018] Lustre: Skipped 1 previous similar message [ 4796.533820] LustreError: 310685:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4796.537741] LustreError: 310685:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4796.599092] Lustre: server umount lustre-OST0000 complete [ 4798.945637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4798.950326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4798.958524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4798.963587] Lustre: Skipped 1 previous similar message [ 4804.623621] LustreError: 310886:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4804.627665] LustreError: 310886:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4804.667860] Lustre: server umount lustre-MDT0000 complete [ 4806.475209] LustreError: 307783:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761967517 with bad export cookie 16615298979534987088 [ 4806.482129] LustreError: 307783:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4806.483432] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4806.664209] Lustre: server umount lustre-MDT0001 complete [ 4810.832250] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 23:25:22 (1761967522) [ 4811.604905] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4812.379199] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 23:25:23 (1761967523) [ 4813.504862] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4817.099558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4818.976043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4820.694285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4822.555687] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4823.080361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4826.380873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4826.515504] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4826.527433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4826.618345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4828.268667] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4832.406380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4832.456209] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4832.471132] Lustre: 313207:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4834.389499] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4836.961500] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4838.810258] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4841.979647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4842.016310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4842.126882] Lustre: Denying initial registration attempt from nid 192.168.202.103@tcp, specified as failover [ 4842.132198] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4842.136845] LustreError: 314809:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 4842.140493] LustreError: 314809:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4842.143578] LustreError: 314809:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4842.170978] Lustre: server umount lustre-OST0000 complete [ 4842.172873] LustreError: 314809:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4848.097380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4848.101906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4848.108699] Lustre: Skipped 1 previous similar message [ 4848.112315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4848.115737] Lustre: Skipped 2 previous similar messages [ 4851.406138] LustreError: 315177:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4851.409802] LustreError: 315177:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4851.443070] Lustre: server umount lustre-MDT0000 complete [ 4853.200180] LustreError: 313197:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761967564 with bad export cookie 16615298979534987956 [ 4853.206819] LustreError: 313197:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4853.207442] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4853.357540] Lustre: server umount lustre-MDT0001 complete [ 4857.771738] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 4859.213164] Key type lgssc unregistered [ 4859.385604] LNet: 315904:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4859.390404] LNetError: 315904:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4859.401572] LNet: Removed LNI 192.168.202.103@tcp [ 4859.839220] Key type .llcrypt unregistered [ 4859.841213] Key type ._llcrypt unregistered [ 4864.014960] Key type ._llcrypt registered [ 4864.016854] Key type .llcrypt registered [ 4864.060418] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4864.400953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4864.435932] alg: No test for adler32 (adler32-zlib) [ 4865.312937] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 4865.408049] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 4866.984165] Key type lgssc registered [ 4867.547782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4871.089070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4872.945491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4874.785631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4876.667835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4877.181267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4885.481820] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4889.660161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4889.671556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4890.805677] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4890.820412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4890.911543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4892.634793] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4896.669084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4896.728852] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4896.747675] Lustre: 318637:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4896.874769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4898.544195] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4900.981862] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4902.653261] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4905.890299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4905.931599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4906.047371] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4906.144938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4908.658117] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4911.198362] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4912.262129] LustreError: 320893:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4912.315281] Lustre: server umount lustre-OST0000 complete [ 4917.217073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4917.220684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4917.225921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4917.730560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4917.739344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4920.272403] LustreError: 321101:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 4920.276410] LustreError: 321101:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4920.312348] Lustre: server umount lustre-MDT0000 complete [ 4922.134464] LustreError: 318627:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761967632 with bad export cookie 4280726011428158327 [ 4922.139805] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4922.141345] LustreError: 318627:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4922.237253] LustreError: 321303:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4922.240623] LustreError: 321303:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4922.294493] Lustre: server umount lustre-MDT0001 complete [ 4927.305937] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 4928.714787] Key type lgssc unregistered [ 4928.868598] LNet: 321827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4928.873138] LNetError: 321827:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4928.883524] LNet: Removed LNI 192.168.202.103@tcp [ 4929.322838] Key type .llcrypt unregistered [ 4929.324909] Key type ._llcrypt unregistered [ 4941.376868] Key type ._llcrypt registered [ 4941.378907] Key type .llcrypt registered [ 4941.428848] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4941.843466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4941.876086] alg: No test for adler32 (adler32-zlib) [ 4942.755076] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 4942.844346] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 4944.432174] Key type lgssc registered [ 4945.002857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4948.897494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4948.908454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4950.164384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4951.977664] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4956.192865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4956.378263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4958.252559] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4960.825751] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4962.460144] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4965.583734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4965.710926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4967.906925] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4970.400360] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4980.195360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4980.199477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4980.202637] Lustre: Skipped 2 previous similar messages [ 4980.208807] Lustre: Skipped 2 previous similar messages [ 4981.196128] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 4984.974349] LustreError: 325429:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4985.008102] Lustre: server umount lustre-MDT0000 complete [ 4985.315300] LustreError: 323189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4985.323915] LustreError: 323189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4986.302525] LustreError: 323189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4986.628992] LustreError: 323174:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761967697 with bad export cookie 5144676694622676144 [ 4986.635267] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4986.635895] LustreError: 323174:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4990.433915] LustreError: 323188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4990.434282] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4990.451561] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4992.782413] LustreError: 325634:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4992.786346] LustreError: 325634:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4992.853172] Lustre: server umount lustre-MDT0001 complete [ 4995.850806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4999.820905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5011.937069] LustreError: 322505:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d5839e45 x1847556993736960/t0(0) o250->MGC192.168.202.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5012.108651] LustreError: 324788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5012.117299] LustreError: 324788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5012.147075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5012.174752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5013.720281] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5016.226419] LustreError: 326443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5016.235935] LustreError: 326443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5017.759284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5017.958390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5019.656190] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5022.058434] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5022.130184] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5027.072449] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5028.677276] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5032.636916] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5037.543624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5037.544834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5037.550713] Lustre: Skipped 1 previous similar message [ 5037.556563] Lustre: Skipped 3 previous similar messages [ 5040.500159] LustreError: 327924:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5040.503954] LustreError: 327924:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5040.560879] Lustre: server umount lustre-OST0000 complete [ 5042.658427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5042.659503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5042.665455] Lustre: Skipped 1 previous similar message [ 5048.463339] LustreError: 328126:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5048.467137] LustreError: 328126:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5048.500452] Lustre: server umount lustre-MDT0000 complete [ 5050.295084] LustreError: 323174:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761967761 with bad export cookie 5144676694622683298 [ 5050.301602] LustreError: 323174:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5050.301791] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5050.459425] Lustre: server umount lustre-MDT0001 complete [ 5055.321513] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5056.746139] Key type lgssc unregistered [ 5056.909634] LNet: 328853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5056.914423] LNetError: 328853:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5056.925543] LNet: Removed LNI 192.168.202.103@tcp [ 5057.367889] Key type .llcrypt unregistered [ 5057.370109] Key type ._llcrypt unregistered [ 5069.532951] Key type ._llcrypt registered [ 5069.534528] Key type .llcrypt registered [ 5069.573292] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5069.877411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5069.890313] alg: No test for adler32 (adler32-zlib) [ 5070.786066] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 5070.881213] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 5072.480177] Key type lgssc registered [ 5073.073195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5076.922497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5076.933343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5078.185547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5080.040254] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5084.282842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5084.476508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5086.275344] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5088.886549] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5090.696762] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5092.834821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5092.835889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5092.842138] Lustre: Skipped 1 previous similar message [ 5094.883367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5094.886809] Lustre: Skipped 2 previous similar messages [ 5098.382484] LustreError: 331599:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5098.424582] Lustre: server umount lustre-MDT0000 complete [ 5100.001937] LustreError: 330221:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5100.010830] LustreError: 330221:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5100.230393] LustreError: 331602:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761967811 with bad export cookie 4525631288300121452 [ 5100.235244] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5100.236956] LustreError: 331602:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5100.344230] LustreError: 331803:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5100.347885] LustreError: 331803:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5100.429608] Lustre: server umount lustre-MDT0001 complete [ 5107.518733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5107.679131] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5107.699118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5107.816485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5109.578623] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5113.817428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5113.871980] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5113.891691] Lustre: 332572:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5114.063346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5115.879472] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5118.541359] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5120.355320] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5124.148038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5124.278664] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5124.284777] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5124.291274] LustreError: 334216:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 5124.296337] LustreError: 334216:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 5124.300765] LustreError: 334216:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 5124.342104] Lustre: server umount lustre-OST0000 complete [ 5124.344937] LustreError: 334216:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5128.544292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5128.666594] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5128.800462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5131.544239] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5134.052617] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5134.897543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5137.580144] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5137.625305] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5137.658203] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5137.700796] Lustre: lustre-OST0001: new disk, initializing [ 5137.702716] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5137.744386] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5140.114181] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5140.913082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5140.919585] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5140.967060] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5142.752351] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5146.088992] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5146.092370] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5146.094053] LustreError: Skipped 1 previous similar message [ 5146.103173] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5146.107338] Lustre: Skipped 1 previous similar message [ 5150.260228] LustreError: 336324:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5150.264266] LustreError: 336324:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5150.298129] Lustre: server umount lustre-OST0001 complete [ 5151.210369] LustreError: 335307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5151.221861] LustreError: 335307:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5155.299722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5155.304322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5155.311474] Lustre: Skipped 1 previous similar message [ 5155.315568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5155.319992] Lustre: Skipped 1 previous similar message [ 5156.327676] LustreError: 334683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5156.335192] LustreError: 334683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5158.258192] LustreError: 336526:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5158.262251] LustreError: 336526:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5158.325626] Lustre: server umount lustre-OST0000 complete [ 5160.417183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5160.421054] LustreError: Skipped 1 previous similar message [ 5160.422974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5160.427969] Lustre: Skipped 1 previous similar message [ 5160.430399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5160.432948] Lustre: Skipped 2 previous similar messages [ 5166.294347] LustreError: 336727:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5166.298062] LustreError: 336727:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5166.346793] Lustre: server umount lustre-MDT0000 complete [ 5166.562669] LustreError: 332583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5166.571986] LustreError: 332583:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5168.186402] LustreError: 334699:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761967878 with bad export cookie 4525631288300121998 [ 5168.192793] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5168.192953] LustreError: 334699:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5168.344863] Lustre: server umount lustre-MDT0001 complete [ 5173.291205] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5174.814932] Key type lgssc unregistered [ 5174.986555] LNet: 337453:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5174.991241] LNetError: 337453:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5174.999624] LNet: Removed LNI 192.168.202.103@tcp [ 5175.485492] Key type .llcrypt unregistered [ 5175.487587] Key type ._llcrypt unregistered [ 5190.925384] Key type ._llcrypt registered [ 5190.927192] Key type .llcrypt registered [ 5190.964294] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 5199.019555] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5199.490603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5199.507010] alg: No test for adler32 (adler32-zlib) [ 5200.355629] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 5200.428556] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 5202.008190] Key type lgssc registered [ 5202.576682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5206.877495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5210.051573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5213.081747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5216.217956] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5219.278558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5219.307110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5219.316291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5220.434509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5220.452434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5220.505039] Lustre: lustre-MDT0000: new disk, initializing [ 5220.542922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5220.554205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5222.257120] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5227.780821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5227.819253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5227.872287] Lustre: 341361:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5227.909042] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5227.912410] Lustre: Skipped 1 previous similar message [ 5227.955037] Lustre: lustre-MDT0001: new disk, initializing [ 5227.999880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5228.015119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5228.019543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5229.662182] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5233.702393] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5235.457865] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5238.741579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5238.777400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5238.935303] Lustre: lustre-OST0000: new disk, initializing [ 5238.938897] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5238.988936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5241.539407] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5244.934211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5244.942726] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5244.995420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5245.594650] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5247.335020] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5247.400972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5248.964480] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5249.026134] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5250.019790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5250.023091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5250.026410] LustreError: Skipped 1 previous similar message [ 5250.034529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5250.040224] Lustre: Skipped 1 previous similar message [ 5251.954139] LustreError: 344253:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5252.014569] Lustre: server umount lustre-OST0000 complete [ 5255.139115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.140188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5255.149183] Lustre: Skipped 2 previous similar messages [ 5259.858432] LustreError: 344454:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5259.862399] LustreError: 344454:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5259.899104] Lustre: server umount lustre-MDT0000 complete [ 5261.705050] LustreError: 341351:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761967972 with bad export cookie 3747646214513312136 [ 5261.710697] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5261.711539] LustreError: 341351:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5261.821318] LustreError: 344655:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5261.824726] LustreError: 344655:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5261.876366] Lustre: server umount lustre-MDT0001 complete [ 5265.976043] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 23:32:57 (1761967977) [ 5268.801461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5271.598145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5282.172687] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 5286.469564] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5291.135658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5294.376688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5297.448484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5300.600642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5303.979626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5304.023251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5304.162954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5304.179335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5304.232312] Lustre: lustre-MDT0000: new disk, initializing [ 5304.270441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5304.282367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5306.116636] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5311.744796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5311.786796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5311.841705] Lustre: 348409:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5311.883616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5311.887321] Lustre: Skipped 1 previous similar message [ 5311.941034] Lustre: lustre-MDT0001: new disk, initializing [ 5312.000317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5312.021458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5312.028266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5313.876475] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5317.924232] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5319.732120] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5323.037184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5323.082384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5323.224600] Lustre: lustre-OST0000: new disk, initializing [ 5323.228876] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5323.280340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5326.017355] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5330.092557] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5332.092653] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5333.007542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5333.013876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5333.083150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5333.251548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5335.169344] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5335.291238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5338.089483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5338.090033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5338.094098] LustreError: Skipped 1 previous similar message [ 5338.102457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5338.107604] Lustre: Skipped 2 previous similar messages [ 5342.322361] LustreError: 351311:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5342.326636] LustreError: 351311:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5342.395160] Lustre: server umount lustre-OST0000 complete [ 5347.809462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5347.814910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5347.822860] Lustre: Skipped 1 previous similar message [ 5347.826796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5350.474934] LustreError: 351512:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5350.479095] LustreError: 351512:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5350.515380] Lustre: server umount lustre-MDT0000 complete [ 5352.071297] LustreError: 349183:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761968062 with bad export cookie 3747646214513313004 [ 5352.074911] LustreError: 349183:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5352.076295] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5352.223783] Lustre: server umount lustre-MDT0001 complete [ 5355.801557] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 23:34:27 (1761968067) [ 5358.524933] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 23:34:29 (1761968069) [ 5361.433157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.621361] LustreError: 352728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5361.628640] LustreError: 352728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5361.650144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5363.381116] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5366.754799] LustreError: 352728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5367.639125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5369.780703] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5372.292884] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5374.099674] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5377.191502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5380.085968] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5382.732347] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5392.026741] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5392.918670] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5393.724486] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5394.559838] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5395.523073] Lustre: Failing over lustre-MDT0000 [ 5395.703428] LustreError: 355148:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5395.705698] LustreError: 355148:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5395.749546] Lustre: server umount lustre-MDT0000 complete [ 5396.965369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5396.970663] LustreError: 354331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5396.972895] Lustre: Skipped 3 previous similar messages [ 5396.981855] LustreError: 354331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5402.082924] LustreError: 352728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5402.091278] LustreError: 352728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5407.205721] LustreError: 352724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5407.215401] LustreError: 352724:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5410.056753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5410.128948] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5410.278449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5410.282795] Lustre: Skipped 2 previous similar messages [ 5410.317505] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5412.067836] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5413.824463] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5415.407970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5415.422708] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5415.454290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5417.182538] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5418.045881] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5420.327925] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5421.257565] Lustre: Failing over lustre-MDT0000 [ 5421.425084] LustreError: 356221:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5421.427925] LustreError: 356221:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5421.473754] Lustre: server umount lustre-MDT0000 complete [ 5424.074218] LustreError: 352724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5424.083772] LustreError: 352724:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5425.639413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5425.641592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5425.646512] Lustre: Skipped 1 previous similar message [ 5435.727037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5435.801860] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5435.875554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 5435.878304] Lustre: Skipped 4 previous similar messages [ 5436.013523] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5437.843907] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5439.424715] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5441.009207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5441.011982] Lustre: Skipped 2 previous similar messages [ 5441.021196] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5441.051176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5442.616255] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5443.388308] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5446.119287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5446.120469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5446.121105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5446.124489] Lustre: Skipped 2 previous similar messages [ 5452.279415] Lustre: server umount lustre-MDT0000 complete [ 5454.004072] LustreError: 352709:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761968164 with bad export cookie 3747646214513315685 [ 5454.010544] LustreError: 352709:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5454.011815] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5454.107690] LustreError: 357444:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5454.110518] LustreError: 357444:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5454.158394] Lustre: server umount lustre-MDT0001 complete [ 5466.098873] Lustre: server umount lustre-OST0000 complete [ 5473.345765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5473.588124] LustreError: 358472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5473.599133] LustreError: 358472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5475.229147] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5479.198596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5479.389503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5479.393631] Lustre: Skipped 2 previous similar messages [ 5480.917870] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5483.378781] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5484.999897] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5488.159932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5490.815483] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5493.302628] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5498.605209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5503.971570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5503.971988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5503.972407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5503.972411] Lustre: Skipped 5 previous similar messages [ 5503.977315] Lustre: Skipped 2 previous similar messages [ 5506.341260] Lustre: server umount lustre-OST0000 complete [ 5522.400153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5522.454739] LustreError: 360898:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5522.458501] LustreError: 360898:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5522.491036] Lustre: server umount lustre-MDT0000 complete [ 5523.427044] LustreError: 359135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5523.435738] LustreError: 359135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5524.244448] LustreError: 358451:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761968235 with bad export cookie 3747646214513316392 [ 5524.250433] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5524.251512] LustreError: 358451:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5524.406249] Lustre: server umount lustre-MDT0001 complete [ 5528.663562] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5530.134835] Key type lgssc unregistered [ 5530.302566] LNet: 361626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5530.307279] LNetError: 361626:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5530.316582] LNet: Removed LNI 192.168.202.103@tcp [ 5530.773787] Key type .llcrypt unregistered [ 5530.775974] Key type ._llcrypt unregistered [ 5543.840164] Key type ._llcrypt registered [ 5543.842038] Key type .llcrypt registered [ 5543.883773] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5544.290128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5544.307232] alg: No test for adler32 (adler32-zlib) [ 5545.155202] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 5545.251921] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 5546.840153] Key type lgssc registered [ 5547.410369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5551.086219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5551.097512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5552.319802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5553.926283] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5557.979438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5558.178405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5559.787985] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5562.438631] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5564.265324] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5567.558200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5567.715456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5570.224826] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5572.735564] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5577.964359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5588.452892] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5588.455468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5588.460464] Lustre: Skipped 2 previous similar messages [ 5588.466601] Lustre: Skipped 2 previous similar messages [ 5593.551059] LustreError: 365337:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5593.573291] LustreError: 363088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5593.581047] LustreError: 363088:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5593.588788] Lustre: server umount lustre-MDT0000 complete [ 5595.330752] LustreError: 363071:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761968306 with bad export cookie 8921926136964611860 [ 5595.336477] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5595.337647] LustreError: 363071:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5595.437075] LustreError: 365541:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5595.439657] LustreError: 365541:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5595.487162] Lustre: server umount lustre-MDT0001 complete [ 5607.545245] LustreError: 365741:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5607.549305] LustreError: 365741:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5607.613629] Lustre: server umount lustre-OST0000 complete [ 5613.979652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5614.182822] LustreError: 366472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5614.211113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5615.790215] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5619.682111] LustreError: 366472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5619.909211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5620.103782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5621.784640] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5624.431654] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5626.190398] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5629.472738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5629.625777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5632.125365] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5634.735450] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5639.851923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5650.406333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5650.408366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5650.413629] Lustre: Skipped 2 previous similar messages [ 5650.417352] Lustre: Skipped 1 previous similar message [ 5653.516885] LustreError: 368796:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5653.520827] LustreError: 368796:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5653.556950] Lustre: server umount lustre-MDT0000 complete [ 5655.309980] LustreError: 366452:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761968366 with bad export cookie 8921926136964612959 [ 5655.314515] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5655.316886] LustreError: 366452:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5655.465663] Lustre: server umount lustre-MDT0001 complete [ 5667.512228] LustreError: 369200:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5667.516290] LustreError: 369200:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5667.579519] Lustre: server umount lustre-OST0000 complete [ 5675.019875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5675.224798] LustreError: 370027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5675.261925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5676.988496] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5680.611991] LustreError: 370027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5681.130887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5683.078625] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5685.656494] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5687.376443] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5690.638122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5693.334089] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5695.956579] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5701.102692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5706.215782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5706.224312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5706.224318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5706.227981] Lustre: Skipped 1 previous similar message [ 5709.105340] LustreError: 372256:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5709.109165] LustreError: 372256:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5709.163618] Lustre: server umount lustre-OST0000 complete [ 5711.331284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5711.332437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5711.338808] Lustre: Skipped 2 previous similar messages [ 5711.345401] Lustre: Skipped 1 previous similar message [ 5716.449917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5716.453655] Lustre: Skipped 1 previous similar message [ 5717.102142] Lustre: server umount lustre-MDT0000 complete [ 5718.875037] LustreError: 370008:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761968429 with bad export cookie 8921926136964613967 [ 5718.880937] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5718.881841] LustreError: 370008:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5719.025458] Lustre: server umount lustre-MDT0001 complete [ 5724.110914] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5725.738644] Key type lgssc unregistered [ 5725.922675] LNet: 373184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5725.926774] LNetError: 373184:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5725.937697] LNet: Removed LNI 192.168.202.103@tcp [ 5726.403428] Key type .llcrypt unregistered [ 5726.405360] Key type ._llcrypt unregistered [ 5739.279204] Key type ._llcrypt registered [ 5739.281305] Key type .llcrypt registered [ 5739.330090] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5739.853503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5739.868690] alg: No test for adler32 (adler32-zlib) [ 5740.746472] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 5740.863173] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 5742.456153] Key type lgssc registered [ 5743.034203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5746.638961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5746.649769] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5746.655179] LustreError: 374572:0:(osd_handler.c:8570:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5746.668551] LustreError: 374572:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5746.673094] LustreError: 374572:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5746.677726] LustreError: 374572:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5746.682952] LustreError: 374572:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5750.285733] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5750.290747] LustreError: 374970:0:(osd_handler.c:8570:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5750.302698] LustreError: 374970:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5750.307258] LustreError: 374970:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5750.311975] LustreError: 374970:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5750.317039] LustreError: 374970:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5759.251540] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5760.749922] Key type lgssc unregistered [ 5760.903701] LNet: 375948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5760.908320] LNetError: 375948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5760.919576] LNet: Removed LNI 192.168.202.103@tcp [ 5761.352637] Key type .llcrypt unregistered [ 5761.354501] Key type ._llcrypt unregistered [ 5769.573760] Key type ._llcrypt registered [ 5769.575328] Key type .llcrypt registered [ 5769.619423] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 5778.036832] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5778.555950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5778.567496] alg: No test for adler32 (adler32-zlib) [ 5779.476212] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 5779.578459] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 5781.184163] Key type lgssc registered [ 5781.748558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5786.051297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5789.180425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5792.180471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5795.278527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5798.645083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5798.674936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5798.684674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5799.818794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5799.840048] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5799.889584] Lustre: lustre-MDT0000: new disk, initializing [ 5799.932742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5799.944440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5801.750058] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5807.459043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5807.501674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5807.557912] Lustre: 379037:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5807.604986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5807.608586] Lustre: Skipped 1 previous similar message [ 5807.657204] Lustre: lustre-MDT0001: new disk, initializing [ 5807.714472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5807.732446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5807.741468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5809.556319] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5813.660271] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5815.377780] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5818.783181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5818.833261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5818.991681] Lustre: lustre-OST0000: new disk, initializing [ 5818.996252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5819.050859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5821.801544] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5825.552751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5825.559205] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5825.624954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5825.921399] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5827.827099] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5827.940332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5829.840629] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5829.958084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5835.750333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5835.752749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5835.754988] LustreError: Skipped 1 previous similar message [ 5835.763219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5835.769134] Lustre: Skipped 1 previous similar message [ 5836.981190] LustreError: 381929:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5837.045723] Lustre: server umount lustre-OST0000 complete [ 5840.868734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5840.870390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5840.876186] Lustre: Skipped 2 previous similar messages [ 5845.073625] LustreError: 382130:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5845.077443] LustreError: 382130:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5845.116442] Lustre: server umount lustre-MDT0000 complete [ 5846.991683] LustreError: 382132:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761968557 with bad export cookie 11306363428876206285 [ 5846.998521] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5847.001116] LustreError: 382132:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5847.111411] LustreError: 382335:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5847.115961] LustreError: 382335:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5847.187677] Lustre: server umount lustre-MDT0001 complete [ 5851.258494] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5852.116679] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 23:42:43 (1761968563) [ 5855.368138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5855.619737] LustreError: 383054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5855.629484] LustreError: 383054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5855.657362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5857.438407] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5860.837059] LustreError: 383054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5861.709900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5861.919385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5863.645192] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5866.232205] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5868.008610] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5871.371099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5871.556832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5874.310509] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5876.938098] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5880.291435] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5880.331965] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5880.420635] Lustre: lustre-OST0001: new disk, initializing [ 5880.424890] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5882.272793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5882.280480] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5882.328463] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5883.118635] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5887.207072] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5892.586490] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5892.587374] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5892.591193] LustreError: Skipped 1 previous similar message [ 5892.593217] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5892.601052] Lustre: Skipped 1 previous similar message [ 5892.603779] Lustre: Skipped 1 previous similar message [ 5895.346406] LustreError: 386227:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5895.350047] LustreError: 386227:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5895.378403] Lustre: server umount lustre-OST0001 complete [ 5897.704929] LustreError: 385669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5897.715891] LustreError: 385669:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5902.831436] LustreError: 384639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5902.841163] LustreError: 384639:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5907.944531] LustreError: 384640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5907.952714] LustreError: 384640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5908.449229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5908.453774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5908.461704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5908.465286] Lustre: Skipped 1 previous similar message [ 5912.545346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5912.549868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5912.557938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5914.357243] LustreError: 386450:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5914.361802] LustreError: 386450:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5914.423470] Lustre: server umount lustre-OST0000 complete [ 5918.181586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5922.513079] LustreError: 386652:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5922.516890] LustreError: 386652:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5922.553892] Lustre: server umount lustre-MDT0000 complete [ 5923.297824] LustreError: 383758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5923.307186] LustreError: 383758:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5924.356704] LustreError: 383767:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761968635 with bad export cookie 11306363428876207153 [ 5924.367454] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5930.598772] Lustre: server umount lustre-MDT0001 complete [ 5935.468671] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5937.059322] Key type lgssc unregistered [ 5937.231645] LNet: 387378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5937.236349] LNetError: 387378:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5937.249716] LNet: Removed LNI 192.168.202.103@tcp [ 5937.721700] Key type .llcrypt unregistered [ 5937.723883] Key type ._llcrypt unregistered [ 5953.555731] Key type ._llcrypt registered [ 5953.557836] Key type .llcrypt registered [ 5953.608925] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 5961.769387] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5962.307813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5962.317416] alg: No test for adler32 (adler32-zlib) [ 5963.190705] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 5963.281359] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 5964.864165] Key type lgssc registered [ 5965.474351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5969.928430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5973.139725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5976.138159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5979.329939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5982.690939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5982.725807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5982.734949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5983.839451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5983.855505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5983.906648] Lustre: lustre-MDT0000: new disk, initializing [ 5983.944606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5983.956818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5985.667552] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5991.189496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5991.226484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5991.284713] Lustre: 391286:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5991.326507] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5991.330125] Lustre: Skipped 1 previous similar message [ 5991.376372] Lustre: lustre-MDT0001: new disk, initializing [ 5991.428546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5991.449373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5991.455131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5993.025537] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5997.031139] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5998.802716] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6002.046346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6002.087438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6002.256624] Lustre: lustre-OST0000: new disk, initializing [ 6002.259516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6002.311101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6005.023373] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6009.114800] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6011.108824] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6011.410082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6011.416872] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6011.478522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6012.261524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6014.101766] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6014.201338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6016.487693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6016.489168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6016.492449] LustreError: Skipped 1 previous similar message [ 6016.497535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6016.502852] Lustre: Skipped 1 previous similar message [ 6017.204168] LustreError: 394190:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6017.266492] Lustre: server umount lustre-OST0000 complete [ 6021.604989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6021.605916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6021.612173] Lustre: Skipped 2 previous similar messages [ 6021.618599] Lustre: Skipped 1 previous similar message [ 6026.722739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6026.726470] Lustre: Skipped 1 previous similar message [ 6031.842617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6031.846308] Lustre: Skipped 1 previous similar message [ 6033.376132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6033.436807] LustreError: 394392:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6033.440659] LustreError: 394392:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6033.478378] Lustre: server umount lustre-MDT0000 complete [ 6035.334608] LustreError: 393019:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761968746 with bad export cookie 6683423096206001129 [ 6035.339532] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6035.341231] LustreError: 393019:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6035.446262] LustreError: 394596:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6035.449764] LustreError: 394596:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6035.529218] Lustre: server umount lustre-MDT0001 complete [ 6039.615688] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 23:45:50 (1761968750) [ 6044.161824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6048.275433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6048.527357] Lustre: lustre-MDT0000: new disk, initializing [ 6048.589902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6050.374344] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6053.858223] LustreError: 395862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6053.867791] LustreError: 395862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6054.614051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6054.792894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6056.670775] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6059.289716] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6061.072217] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6064.100210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6064.101347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6064.110275] Lustre: Skipped 1 previous similar message [ 6075.362459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6075.366907] Lustre: Skipped 6 previous similar messages [ 6076.384158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6076.440416] LustreError: 397190:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6076.444281] LustreError: 397190:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6076.480548] Lustre: server umount lustre-MDT0000 complete [ 6078.271299] LustreError: 395841:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761968789 with bad export cookie 6683423096206001997 [ 6078.281471] LustreError: 395841:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6078.282112] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6078.440202] Lustre: server umount lustre-MDT0001 complete [ 6082.964531] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 23:46:34 (1761968794) [ 6086.224105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6086.457117] LustreError: 398116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6086.491658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6088.403272] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6091.746407] LustreError: 398116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6092.643301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6094.567060] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6097.152146] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6098.889087] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6102.096672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6104.649485] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6107.242885] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6115.927456] Lustre: Permanently deactivating lustre-OST0000 [ 6115.936770] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6125.563564] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6126.707211] LustreError: 400884:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6126.714032] LustreError: 400884:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 6126.718462] LustreError: 400884:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6127.544424] LustreError: 400932:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 6127.549145] LustreError: 400932:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6134.516107] LustreError: 401030:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6134.520062] LustreError: 401030:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6134.576049] Lustre: server umount lustre-OST0000 complete [ 6135.879175] LustreError: 401182:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 6135.883803] LustreError: 401182:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6138.339271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6138.340142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6138.346544] Lustre: Skipped 1 previous similar message [ 6151.136147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6151.192203] LustreError: 401279:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6151.195888] LustreError: 401279:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6151.228342] Lustre: server umount lustre-MDT0000 complete [ 6152.990035] LustreError: 398095:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761968863 with bad export cookie 6683423096206002536 [ 6152.995195] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6152.996973] LustreError: 398095:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6162.473355] Lustre: server umount lustre-MDT0001 complete [ 6166.582454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6168.197691] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6169.476873] LustreError: 402317:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6172.428907] Lustre: 402658:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.103@tcp [ 6173.395485] LustreError: 402755:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6173.399631] LustreError: 402755:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6173.440404] Lustre: server umount MGS complete [ 6178.903754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6179.140362] LustreError: 403384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6179.145689] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6179.150621] Lustre: Skipped 1 previous similar message [ 6179.179528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6179.183148] Lustre: Skipped 2 previous similar messages [ 6180.824882] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6184.417822] LustreError: 403384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6184.902740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6185.059915] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6186.828304] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6189.472509] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6191.227573] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6192.158950] Lustre: Permanently reactivating lustre-OST0000 [ 6192.167980] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6192.172160] Lustre: Skipped 2 previous similar messages [ 6215.648261] LustreError: 404833:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6220.063222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6220.256929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6220.260870] Lustre: Skipped 1 previous similar message [ 6222.717549] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6225.340643] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6234.593275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6234.597941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6234.605990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6234.609805] Lustre: Skipped 5 previous similar messages [ 6235.623539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6238.707108] LustreError: 406580:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6238.710940] LustreError: 406580:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6238.768392] Lustre: server umount lustre-OST0000 complete [ 6246.703616] Lustre: server umount lustre-MDT0000 complete [ 6248.515295] LustreError: 403365:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761968959 with bad export cookie 6683423096206003481 [ 6248.520325] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6248.522221] LustreError: 403365:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6248.681161] Lustre: server umount lustre-MDT0001 complete [ 6253.729333] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6255.194162] Key type lgssc unregistered [ 6255.367677] LNet: 407512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6255.372304] LNetError: 407512:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6255.383631] LNet: Removed LNI 192.168.202.103@tcp [ 6255.830689] Key type .llcrypt unregistered [ 6255.832796] Key type ._llcrypt unregistered [ 6264.067085] Key type ._llcrypt registered [ 6264.068859] Key type .llcrypt registered [ 6264.114651] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 6272.295539] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6272.650730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6272.658750] alg: No test for adler32 (adler32-zlib) [ 6273.586154] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 6273.688421] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6275.288128] Key type lgssc registered [ 6275.946837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6280.515694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6283.816048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6286.868576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6290.085345] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6293.498862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6293.533923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6293.543503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6294.661599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6294.681370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6294.734827] Lustre: lustre-MDT0000: new disk, initializing [ 6294.773323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6294.782135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6296.600736] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6302.360980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6302.402225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6302.464848] Lustre: 410604:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6302.513600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6302.517162] Lustre: Skipped 1 previous similar message [ 6302.568757] Lustre: lustre-MDT0001: new disk, initializing [ 6302.624537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6302.647010] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6302.653015] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6304.427885] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6308.402325] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6310.216976] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6313.593913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6313.638437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6313.803972] Lustre: lustre-OST0000: new disk, initializing [ 6313.807963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6313.863177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6316.579622] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6320.145769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6320.152566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6320.210668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6320.605561] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6322.630487] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6322.751241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6324.585229] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6324.702241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6330.345636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6330.346144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6330.359639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6330.363260] Lustre: Skipped 1 previous similar message [ 6331.635198] LustreError: 413499:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6331.703145] Lustre: server umount lustre-OST0000 complete [ 6335.460553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6335.462342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6335.467982] Lustre: Skipped 2 previous similar messages [ 6339.664853] LustreError: 413701:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6339.668832] LustreError: 413701:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6339.720472] Lustre: server umount lustre-MDT0000 complete [ 6341.568052] LustreError: 410596:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761969052 with bad export cookie 14574501870226930851 [ 6341.574962] LustreError: 410596:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6341.577139] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6341.684650] LustreError: 413903:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6341.689262] LustreError: 413903:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6341.757502] Lustre: server umount lustre-MDT0001 complete [ 6345.768781] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6346.656852] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 23:50:57 (1761969057) [ 6350.023360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6350.298283] LustreError: 414622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6350.309594] LustreError: 414622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6350.342612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6352.162978] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6355.426641] LustreError: 414622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6356.383180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6356.603666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6358.349115] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6360.941761] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6362.658215] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6365.978940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6366.158589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6368.921056] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6371.576683] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6372.581153] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6378.564215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6380.517111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6380.519330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6380.525890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6380.529850] LustreError: Skipped 1 previous similar message [ 6380.533329] Lustre: Skipped 1 previous similar message [ 6385.635911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6385.637845] Lustre: Skipped 1 previous similar message [ 6386.418192] LustreError: 417042:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6386.422770] LustreError: 417042:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6386.485537] Lustre: server umount lustre-OST0000 complete [ 6390.756804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6390.758563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6390.765368] Lustre: Skipped 2 previous similar messages [ 6390.771481] Lustre: Skipped 1 previous similar message [ 6394.510571] LustreError: 417243:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6394.514471] LustreError: 417243:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6394.566597] Lustre: server umount lustre-MDT0000 complete [ 6395.874528] LustreError: 414640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6395.883894] LustreError: 414640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6396.441372] LustreError: 414602:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761969107 with bad export cookie 14574501870226931733 [ 6396.449516] LustreError: 414602:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6396.450060] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6396.633674] Lustre: server umount lustre-MDT0001 complete [ 6401.763289] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6403.404640] Key type lgssc unregistered [ 6403.588746] LNet: 417969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6403.594333] LNetError: 417969:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6403.606724] LNet: Removed LNI 192.168.202.103@tcp [ 6404.127726] Key type .llcrypt unregistered [ 6404.130290] Key type ._llcrypt unregistered [ 6414.786651] Key type ._llcrypt registered [ 6414.788234] Key type .llcrypt registered [ 6414.844838] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6414.993310] Key type .llcrypt unregistered [ 6414.995256] Key type ._llcrypt unregistered [ 6424.224436] Key type ._llcrypt registered [ 6424.226167] Key type .llcrypt registered [ 6424.276466] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6424.724712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6424.794759] alg: No test for adler32 (adler32-zlib) [ 6425.724730] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 6425.859581] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6427.480184] Key type lgssc registered [ 6428.133825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6432.205080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6432.216592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6433.460629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6435.448814] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6439.766671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6439.952125] LustreError: 420097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6439.958841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6439.960456] LustreError: 420097:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6442.690105] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6444.070343] LustreError: 420103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6444.080111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6444.080365] LustreError: 420103:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6445.414354] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6448.800139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6449.044458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6450.822638] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6457.514069] Lustre: 420097:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 6457.520723] Lustre: 420097:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6457.525691] Lustre: 420097:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 6457.530665] Lustre: 420097:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/153/0, punch: 0/0/0, quota 1/3/0 [ 6457.535507] Lustre: 420097:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6457.539983] Lustre: 420097:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6459.370119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6459.370319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6459.373353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6459.377357] Lustre: Skipped 1 previous similar message [ 6464.491518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6464.495211] Lustre: Skipped 2 previous similar messages [ 6465.076284] LustreError: 422139:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6465.135533] Lustre: server umount lustre-OST0000 complete [ 6469.092406] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6469.096771] LustreError: Skipped 1 previous similar message [ 6469.099385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6469.107399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6473.106567] LustreError: 422340:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6473.110237] LustreError: 422340:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6473.155781] Lustre: server umount lustre-MDT0000 complete [ 6474.722623] LustreError: 420103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6474.731895] LustreError: 420103:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6474.988143] LustreError: 420085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761969185 with bad export cookie 4030478463862195578 [ 6474.997195] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6475.097429] LustreError: 422542:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6475.099907] LustreError: 422542:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6475.179975] Lustre: server umount lustre-MDT0001 complete [ 6480.128601] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6481.564821] Key type lgssc unregistered [ 6481.706676] LNet: 423066:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6481.711471] LNetError: 423066:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6481.723694] LNet: Removed LNI 192.168.202.103@tcp [ 6482.157124] Key type .llcrypt unregistered [ 6482.159182] Key type ._llcrypt unregistered [ 6494.326980] Key type ._llcrypt registered [ 6494.329218] Key type .llcrypt registered [ 6494.370595] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6494.724349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6494.758765] alg: No test for adler32 (adler32-zlib) [ 6495.629490] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 6495.714839] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6497.296198] Key type lgssc registered [ 6497.869312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6501.497911] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6501.509376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6517.024350] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6522.144189] LustreError: 424406:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 6548.768321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6551.215669] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6553.663452] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6556.926709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6557.210262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6557.247236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6558.982242] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6562.276306] LustreError: 424433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6562.285910] LustreError: 424433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6563.151804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6563.332521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6565.031654] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6567.526286] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6569.142106] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6570.515118] Lustre: 425320:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 6570.521937] Lustre: 425320:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6570.526491] Lustre: 425320:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 6570.531478] Lustre: 425320:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 6570.536443] Lustre: 425320:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6570.540992] Lustre: 425320:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6572.513533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6572.518054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6572.525888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6573.541171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6573.544559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6573.545743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6578.096129] LustreError: 426666:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 6578.151328] Lustre: server umount lustre-OST0000 complete [ 6581.732466] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6581.737094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6581.745181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6581.748781] Lustre: Skipped 1 previous similar message [ 6582.030381] LustreError: 426870:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6582.034300] LustreError: 426870:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6582.071711] Lustre: server umount lustre-MDT0000 complete [ 6583.779221] LustreError: 425326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6583.781366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6583.783498] LustreError: 424413:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761969294 with bad export cookie 13456594350711882506 [ 6583.783804] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6583.808396] LustreError: 423745:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c33b1063 x1847558622177664/t0(0) o38->lustre-MDT0000-lwp-MDT0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6583.909482] LustreError: 427072:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6583.911991] LustreError: 427072:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6583.967952] Lustre: server umount lustre-MDT0001 complete [ 6588.462788] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6589.833681] Key type lgssc unregistered [ 6590.005618] LNet: 427596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6590.010288] LNetError: 427596:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6590.020649] LNet: Removed LNI 192.168.202.103@tcp [ 6590.453071] Key type .llcrypt unregistered [ 6590.455154] Key type ._llcrypt unregistered [ 6602.506490] Key type ._llcrypt registered [ 6602.508602] Key type .llcrypt registered [ 6602.558362] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6602.882647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6602.933257] alg: No test for adler32 (adler32-zlib) [ 6603.836919] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 6603.976220] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6605.552221] Key type lgssc registered [ 6606.077951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6609.634486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6609.646095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6610.834867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6612.572915] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6616.627603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6616.818192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6618.694423] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6621.354485] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6623.182760] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6626.458435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6626.645895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6629.096717] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6631.404432] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6635.886106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6641.125203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6641.126781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6641.132440] Lustre: Skipped 2 previous similar messages [ 6641.138501] Lustre: Skipped 2 previous similar messages [ 6642.122224] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 6644.110395] LustreError: 431186:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6644.145627] Lustre: server umount lustre-MDT0000 complete [ 6645.203908] LustreError: 428942:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.3@tcp arrived at 1761969356 with bad export cookie 15932408994159497112 [ 6645.210469] LustreError: 428942:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6646.169073] LustreError: 428942:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761969356 with bad export cookie 15932408994159496461 [ 6646.178449] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6646.242118] LustreError: 428962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6646.242850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6646.251045] LustreError: 428962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6646.259213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6651.361887] LustreError: 428962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6651.362356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6651.370837] LustreError: 428962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6652.276175] LustreError: 431389:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6652.279461] LustreError: 431389:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6652.340410] Lustre: server umount lustre-OST0000 complete [ 6655.459148] LustreError: 428962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6655.472331] LustreError: 428962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6655.640109] LustreError: 431743:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6655.642578] LustreError: 431743:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6655.720845] Lustre: server umount lustre-MDT0001 complete [ 6660.732281] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6662.326232] Key type lgssc unregistered [ 6662.514733] LNet: 432268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6662.520994] LNetError: 432268:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6662.531753] LNet: Removed LNI 192.168.202.103@tcp [ 6663.003067] Key type .llcrypt unregistered [ 6663.005129] Key type ._llcrypt unregistered [ 6675.240934] Key type ._llcrypt registered [ 6675.242188] Key type .llcrypt registered [ 6675.271162] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6675.692428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6675.701592] alg: No test for adler32 (adler32-zlib) [ 6676.560348] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 6676.646769] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6678.232164] Key type lgssc registered [ 6678.861385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6682.559291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6682.570128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6683.764982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6685.217769] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6689.235002] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6689.422871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6691.179862] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6693.761381] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6695.475200] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6698.709080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6698.899768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6701.472177] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6704.110498] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6709.165474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6714.341909] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6714.342198] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6714.355814] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6714.359538] Lustre: Skipped 1 previous similar message [ 6719.459337] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6719.462989] Lustre: Skipped 2 previous similar messages [ 6724.578324] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6724.581887] Lustre: Skipped 2 previous similar messages [ 6725.088166] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6725.143558] LustreError: 435859:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6725.170507] Lustre: server umount lustre-MDT0001 complete [ 6729.699220] LustreError: 434351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6729.700265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6729.700543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6729.700556] Lustre: Skipped 1 previous similar message [ 6729.701865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6729.701872] Lustre: Skipped 1 previous similar message [ 6729.708225] LustreError: 434351:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6733.556126] LustreError: 436065:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6733.560086] LustreError: 436065:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6733.615401] Lustre: server umount lustre-OST0000 complete [ 6734.818054] LustreError: 434351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6735.462861] LustreError: 436266:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6735.466313] LustreError: 436266:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6735.531343] Lustre: server umount lustre-MDT0000 complete [ 6741.748563] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6743.227289] Key type lgssc unregistered [ 6743.399753] LNet: 436943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6743.404474] LNetError: 436943:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6743.415563] LNet: Removed LNI 192.168.202.103@tcp [ 6743.873379] Key type .llcrypt unregistered [ 6743.875479] Key type ._llcrypt unregistered [ 6754.987880] Key type ._llcrypt registered [ 6754.989612] Key type .llcrypt registered [ 6755.025550] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 6762.951909] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6763.328744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6763.342142] alg: No test for adler32 (adler32-zlib) [ 6764.218829] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 6764.334254] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6765.944181] Key type lgssc registered [ 6766.578693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6771.035323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6774.217567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6777.208558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6780.349550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6784.816235] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6789.087270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6792.355409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6792.392795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6792.401919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6793.516094] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6793.534368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6793.594193] Lustre: lustre-MDT0000: new disk, initializing [ 6793.637496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6793.650265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6795.534859] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6800.307621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6803.614633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6803.659301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6803.805332] Lustre: lustre-OST0000: new disk, initializing [ 6803.808927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6803.812309] Lustre: Skipped 1 previous similar message [ 6803.855656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6805.602066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6805.608210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6805.623493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6806.602388] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6810.719564] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6815.115051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6818.449923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6818.491667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6818.551380] Lustre: 440865:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6818.604103] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6818.658102] Lustre: lustre-MDT0001: new disk, initializing [ 6818.731171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6818.756095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6818.762599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6820.564417] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6825.222889] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6825.320424] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6827.168376] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6827.293185] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6830.055647] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6830.058685] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6830.065186] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6830.071436] Lustre: Skipped 1 previous similar message [ 6834.636685] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.3@tcp (stopping) [ 6835.727516] LustreError: 444245:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6835.763313] Lustre: server umount lustre-MDT0001 complete [ 6845.414248] LustreError: 440873:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6845.423686] LustreError: 440873:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6850.532997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6850.533172] LustreError: 443093:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6850.539982] Lustre: Skipped 1 previous similar message [ 6850.552782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6850.556471] Lustre: Skipped 2 previous similar messages [ 6855.651897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6855.652878] LustreError: 440877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6860.771701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6860.773431] LustreError: 440877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6863.328154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6863.357152] LustreError: 444519:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6863.361182] LustreError: 444519:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6863.428525] Lustre: server umount lustre-OST0000 complete [ 6871.509819] LustreError: 444722:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6871.513752] LustreError: 444722:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6871.615829] Lustre: server umount lustre-MDT0000 complete [ 6878.025686] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6879.627687] Key type lgssc unregistered [ 6879.801593] LNet: 445399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6879.806531] LNetError: 445399:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6879.818704] LNet: Removed LNI 192.168.202.103@tcp [ 6880.311484] Key type .llcrypt unregistered [ 6880.313247] Key type ._llcrypt unregistered [ 6899.239679] Key type ._llcrypt registered [ 6899.241819] Key type .llcrypt registered [ 6899.291463] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6899.463565] Key type .llcrypt unregistered [ 6899.465553] Key type ._llcrypt unregistered [ 6908.385742] Key type ._llcrypt registered [ 6908.387663] Key type .llcrypt registered [ 6908.439068] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6908.982662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6909.017831] alg: No test for adler32 (adler32-zlib) [ 6909.920385] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 6910.035682] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6911.656218] Key type lgssc registered [ 6912.271079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6916.353360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6919.577837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6922.668109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6926.021358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6926.058346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6926.067167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6927.170955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6927.185636] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6927.220910] Lustre: lustre-MDT0000: new disk, initializing [ 6927.260686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6927.268849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6929.072916] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6934.766207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6934.797043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6934.854046] Lustre: 448706:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6934.900186] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6934.904018] Lustre: Skipped 1 previous similar message [ 6934.952805] Lustre: lustre-MDT0001: new disk, initializing [ 6934.993260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6935.010882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6935.015436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6936.816553] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6940.985462] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6942.708553] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6946.081098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6946.128737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6946.293264] Lustre: lustre-OST0000: new disk, initializing [ 6946.296960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6946.353040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6949.034384] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6949.714442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6949.720889] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6949.779935] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6953.175958] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6960.101935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6960.104914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6960.108850] Lustre: Skipped 1 previous similar message [ 6960.115021] Lustre: Skipped 2 previous similar messages [ 6962.260186] LustreError: 451139:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6962.309185] Lustre: server umount lustre-MDT0000 complete [ 6964.157591] LustreError: 448696:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761969674 with bad export cookie 9706516113411721445 [ 6964.163709] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6964.164919] LustreError: 448696:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6964.262516] LustreError: 451342:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6964.265754] LustreError: 451342:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6964.332197] Lustre: server umount lustre-MDT0001 complete [ 6975.352503] LustreError: 451542:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6975.356344] LustreError: 451542:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6975.431466] Lustre: server umount lustre-OST0000 complete [ 6980.518906] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:01:31 (1761969691) [ 6981.046631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6984.410606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6984.686563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6986.519263] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6989.794435] LustreError: 452371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6989.803948] LustreError: 452371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6990.727954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6990.927284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6992.778994] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6995.470386] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6997.227898] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7000.588284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7000.635897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7000.758310] Lustre: Found index 0 for lustre-OST0000, updating log [ 7000.876216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7003.631774] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7006.286774] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7010.099736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7015.397217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7015.401991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7015.405224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7015.409358] Lustre: Skipped 2 previous similar messages [ 7016.485837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7016.489510] Lustre: Skipped 2 previous similar messages [ 7019.379173] LustreError: 454608:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7019.382996] LustreError: 454608:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7019.442724] Lustre: server umount lustre-OST0000 complete [ 7020.822366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7022.820452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7022.829077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7022.832688] Lustre: Skipped 1 previous similar message [ 7028.194796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7028.198416] Lustre: Skipped 1 previous similar message [ 7036.384193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7036.443806] LustreError: 454863:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7036.447689] LustreError: 454863:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7036.492819] Lustre: server umount lustre-MDT0000 complete [ 7038.345237] LustreError: 453975:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761969749 with bad export cookie 9706516113411722838 [ 7038.352327] LustreError: 453975:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7038.354672] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7038.504110] Lustre: server umount lustre-MDT0001 complete [ 7042.765213] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 00:02:34 (1761969754) [ 7043.291053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7043.803955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7044.318408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7044.815498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7045.314450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7045.821180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7046.331669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7046.845145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7047.327580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7047.836178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7048.348464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7048.807094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7049.312480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7049.808281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7050.316739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7050.806740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7051.315368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7051.823343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7052.328963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7052.824905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7053.331162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7053.829288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7054.327996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7054.835249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7055.336201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7055.842747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7056.349341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7056.834848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7057.334097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7057.835188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7058.329382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7058.828223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7059.316607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7059.808801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7060.294709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7060.800941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7061.301299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7061.797965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7062.290610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7062.792902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7063.290059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7063.787557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7064.294870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7064.795261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7065.265145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7065.760098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7066.249379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7066.744500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7067.236820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7067.739727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7068.243895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7072.057663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7072.304203] LustreError: 458453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7072.347481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7074.215334] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7077.347308] LustreError: 458453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7078.453271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7080.464384] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7083.175415] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7085.032970] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7088.369211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7088.423064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7088.560497] Lustre: Found index 0 for lustre-OST0000, updating log [ 7091.445961] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7094.128425] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7096.294508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7096.301919] Lustre: Skipped 1 previous similar message [ 7096.304791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7096.305478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7096.313089] Lustre: Skipped 2 previous similar messages [ 7097.200164] LustreError: 460740:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7097.204104] LustreError: 460740:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7097.266470] Lustre: server umount lustre-OST0000 complete [ 7099.361420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7100.390677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7100.398070] Lustre: Skipped 1 previous similar message [ 7105.333183] Lustre: server umount lustre-MDT0000 complete [ 7105.507093] LustreError: 459552:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7105.517695] LustreError: 459552:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7107.171201] LustreError: 458433:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761969817 with bad export cookie 9706516113411723909 [ 7107.175861] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7107.178461] LustreError: 458433:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7107.332267] Lustre: server umount lustre-MDT0001 complete [ 7108.717616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7116.428165] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 7121.262398] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7125.819941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7129.172143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7132.301673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7135.609508] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7139.049420] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 00:04:10 (1761969850) [ 7147.485163] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7148.955718] Key type lgssc unregistered [ 7151.200886] LNet: 464690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7151.205674] LNetError: 464690:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7263.849147] LNet: Removed LNI 192.168.202.103@tcp [ 7264.309058] Key type .llcrypt unregistered [ 7264.311041] Key type ._llcrypt unregistered [ 7273.286192] Key type ._llcrypt registered [ 7273.287663] Key type .llcrypt registered [ 7273.347279] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7273.777033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7273.807345] alg: No test for adler32 (adler32-zlib) [ 7274.711456] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 7274.834231] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 7276.424184] Key type lgssc registered [ 7277.079819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7279.393826] LNet: Added LNI 192.168.202.103@tcp100 [8/256/0/180] [ 7279.838417] LNet: Added LNI 192.168.202.103@tcp101 [8/256/0/180] [ 7280.281737] LNet: Added LNI 192.168.202.103@tcp102 [8/256/0/180] [ 7280.717849] LNet: Added LNI 192.168.202.103@tcp103 [8/256/0/180] [ 7281.153610] LNet: Added LNI 192.168.202.103@tcp104 [8/256/0/180] [ 7281.594814] LNet: Added LNI 192.168.202.103@tcp105 [8/256/0/180] [ 7282.038716] LNet: Added LNI 192.168.202.103@tcp106 [8/256/0/180] [ 7282.483774] LNet: Added LNI 192.168.202.103@tcp107 [8/256/0/180] [ 7282.922916] LNet: Added LNI 192.168.202.103@tcp108 [8/256/0/180] [ 7283.359507] LNet: Added LNI 192.168.202.103@tcp109 [8/256/0/180] [ 7283.798604] LNet: Added LNI 192.168.202.103@tcp110 [8/256/0/180] [ 7284.237649] LNet: Added LNI 192.168.202.103@tcp111 [8/256/0/180] [ 7284.669064] LNet: Added LNI 192.168.202.103@tcp112 [8/256/0/180] [ 7285.105581] LNet: Added LNI 192.168.202.103@tcp113 [8/256/0/180] [ 7285.528774] LNet: Added LNI 192.168.202.103@tcp114 [8/256/0/180] [ 7285.973708] LNet: Added LNI 192.168.202.103@tcp115 [8/256/0/180] [ 7286.413755] LNet: Added LNI 192.168.202.103@tcp116 [8/256/0/180] [ 7286.851780] LNet: Added LNI 192.168.202.103@tcp117 [8/256/0/180] [ 7287.271678] LNet: Added LNI 192.168.202.103@tcp118 [8/256/0/180] [ 7287.707218] LNet: Added LNI 192.168.202.103@tcp119 [8/256/0/180] [ 7288.148927] LNet: Added LNI 192.168.202.103@tcp120 [8/256/0/180] [ 7288.564134] LNet: Added LNI 192.168.202.103@tcp121 [8/256/0/180] [ 7288.998184] LNet: Added LNI 192.168.202.103@tcp122 [8/256/0/180] [ 7289.428869] LNet: Added LNI 192.168.202.103@tcp123 [8/256/0/180] [ 7289.814509] LNet: Added LNI 192.168.202.103@tcp124 [8/256/0/180] [ 7290.254931] LNet: Added LNI 192.168.202.103@tcp125 [8/256/0/180] [ 7290.679085] LNet: Added LNI 192.168.202.103@tcp126 [8/256/0/180] [ 7291.103254] LNet: Added LNI 192.168.202.103@tcp127 [8/256/0/180] [ 7291.538787] LNet: Added LNI 192.168.202.103@tcp128 [8/256/0/180] [ 7291.979260] LNet: Added LNI 192.168.202.103@tcp129 [8/256/0/180] [ 7292.416980] LNet: Added LNI 192.168.202.103@tcp130 [8/256/0/180] [ 7292.856637] LNet: Added LNI 192.168.202.103@tcp131 [8/256/0/180] [ 7293.280852] LNet: Added LNI 192.168.202.103@tcp132 [8/256/0/180] [ 7293.697532] LNet: Added LNI 192.168.202.103@tcp133 [8/256/0/180] [ 7294.137192] LNet: Added LNI 192.168.202.103@tcp134 [8/256/0/180] [ 7294.569436] LNet: Added LNI 192.168.202.103@tcp135 [8/256/0/180] [ 7297.857120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7297.892838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7297.902067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7298.962980] Lustre: 467669:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.103@tcp: can't add NID 192.168.202.103@tcp130: rc = -75 [ 7299.060871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7299.081865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7299.132978] Lustre: lustre-MDT0000: new disk, initializing [ 7299.187229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7299.199467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7301.121431] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7306.774277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7306.820384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7306.890985] Lustre: 467688:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7306.944791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7306.948206] Lustre: Skipped 1 previous similar message [ 7306.998787] Lustre: lustre-MDT0001: new disk, initializing [ 7307.016948] Lustre: 468461:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.103@tcp: can't add NID 192.168.202.103@tcp130: rc = -75 [ 7307.022752] Lustre: 468461:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7307.092178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7307.115183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7307.121757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7308.982645] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7312.904814] Lustre: 468923:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.103@tcp: can't add NID 192.168.202.103@tcp130: rc = -75 [ 7312.908959] Lustre: 468923:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7313.109660] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7314.916455] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7318.322392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7318.376303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7318.571547] Lustre: lustre-OST0000: new disk, initializing [ 7318.574811] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7318.638164] Lustre: 469426:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.103@tcp: can't add NID 192.168.202.103@tcp130: rc = -75 [ 7318.643815] Lustre: 469426:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 7318.668935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7321.440485] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7322.684085] Lustre: 469856:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.103@tcp: can't add NID 192.168.202.103@tcp130: rc = -75 [ 7322.689917] Lustre: 469856:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7322.723950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7322.730405] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7322.817129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7325.539407] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7333.349751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7333.354295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7333.354364] LustreError: Skipped 1 previous similar message [ 7333.355791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7333.361276] Lustre: Skipped 1 previous similar message [ 7335.090189] LustreError: 470254:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7335.150895] Lustre: server umount lustre-OST0000 complete [ 7337.953527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7337.958194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7337.966455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7337.970190] Lustre: Skipped 1 previous similar message [ 7343.586963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7343.590499] Lustre: Skipped 3 previous similar messages [ 7348.706952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7348.710418] Lustre: Skipped 1 previous similar message [ 7351.264150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7351.321940] LustreError: 470457:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7351.325818] LustreError: 470457:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7351.378103] Lustre: server umount lustre-MDT0000 complete [ 7353.207849] LustreError: 467680:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761970064 with bad export cookie 12686302155982478870 [ 7353.217118] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7353.316087] LustreError: 470660:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7353.319440] LustreError: 470660:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7353.387956] Lustre: server umount lustre-MDT0001 complete [ 7358.352968] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7359.923628] Key type lgssc unregistered [ 7360.143936] LNet: 471192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7360.149597] LNetError: 471192:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7360.154172] LNet: Removed LNI 192.168.202.103@tcp [ 7361.184188] LNet: Removed LNI 192.168.202.103@tcp100 [ 7361.186684] LNetError: 471192:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7361.190920] LNetError: 471192:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7361.195512] LNet: Removed LNI 192.168.202.103@tcp101 [ 7361.197854] LNet: Removed LNI 192.168.202.103@tcp102 [ 7361.200231] LNet: Removed LNI 192.168.202.103@tcp103 [ 7361.202422] LNet: Removed LNI 192.168.202.103@tcp104 [ 7361.204743] LNet: Removed LNI 192.168.202.103@tcp105 [ 7361.207141] LNet: Removed LNI 192.168.202.103@tcp106 [ 7361.209462] LNet: Removed LNI 192.168.202.103@tcp107 [ 7361.211841] LNet: Removed LNI 192.168.202.103@tcp108 [ 7361.214181] LNet: Removed LNI 192.168.202.103@tcp109 [ 7361.216508] LNet: Removed LNI 192.168.202.103@tcp110 [ 7361.218855] LNet: Removed LNI 192.168.202.103@tcp111 [ 7361.221200] LNet: Removed LNI 192.168.202.103@tcp112 [ 7361.223548] LNet: Removed LNI 192.168.202.103@tcp113 [ 7361.225909] LNet: Removed LNI 192.168.202.103@tcp114 [ 7361.228243] LNet: Removed LNI 192.168.202.103@tcp115 [ 7361.230444] LNet: Removed LNI 192.168.202.103@tcp116 [ 7361.232523] LNet: Removed LNI 192.168.202.103@tcp117 [ 7361.234560] LNet: Removed LNI 192.168.202.103@tcp118 [ 7361.236545] LNet: Removed LNI 192.168.202.103@tcp119 [ 7361.238478] LNet: Removed LNI 192.168.202.103@tcp120 [ 7361.240415] LNet: Removed LNI 192.168.202.103@tcp121 [ 7361.242279] LNet: Removed LNI 192.168.202.103@tcp122 [ 7361.244108] LNet: Removed LNI 192.168.202.103@tcp123 [ 7361.245912] LNet: Removed LNI 192.168.202.103@tcp124 [ 7361.247691] LNet: Removed LNI 192.168.202.103@tcp125 [ 7361.249377] LNet: Removed LNI 192.168.202.103@tcp126 [ 7361.250863] LNet: Removed LNI 192.168.202.103@tcp127 [ 7361.252412] LNet: Removed LNI 192.168.202.103@tcp128 [ 7361.253950] LNet: Removed LNI 192.168.202.103@tcp129 [ 7361.255431] LNet: Removed LNI 192.168.202.103@tcp130 [ 7361.256887] LNet: Removed LNI 192.168.202.103@tcp131 [ 7361.258256] LNet: Removed LNI 192.168.202.103@tcp132 [ 7361.259587] LNet: Removed LNI 192.168.202.103@tcp133 [ 7361.260880] LNet: Removed LNI 192.168.202.103@tcp134 [ 7361.268441] LNet: Removed LNI 192.168.202.103@tcp135 [ 7361.528795] Key type .llcrypt unregistered [ 7361.530251] Key type ._llcrypt unregistered [ 7369.430281] Key type ._llcrypt registered [ 7369.431726] Key type .llcrypt registered [ 7369.473030] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 7377.011888] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7377.493997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7377.501381] alg: No test for adler32 (adler32-zlib) [ 7378.373779] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 7378.474566] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 7380.072230] Key type lgssc registered [ 7380.617705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7384.345453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7387.140550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7389.769571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7392.631460] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7395.401861] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 00:08:26 (1761970106) [ 7401.356016] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7402.400728] Key type lgssc unregistered [ 7402.541459] LNet: 475182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7402.543354] LNetError: 475182:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7402.552390] LNet: Removed LNI 192.168.202.103@tcp [ 7402.886498] Key type .llcrypt unregistered [ 7402.887941] Key type ._llcrypt unregistered [ 7409.756386] Key type ._llcrypt registered [ 7409.757557] Key type .llcrypt registered [ 7409.789883] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7410.197328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7410.225163] alg: No test for adler32 (adler32-zlib) [ 7411.091171] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 7411.186090] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 7412.768161] Key type lgssc registered [ 7413.159587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7416.456908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7416.478173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7416.482814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7417.596670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7417.613862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7417.666764] Lustre: lustre-MDT0000: new disk, initializing [ 7417.697050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7417.706126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7419.046329] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7423.184908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7423.207285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7423.242063] Lustre: 476345:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7423.267974] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7423.271079] Lustre: Skipped 1 previous similar message [ 7423.319058] Lustre: lustre-MDT0001: new disk, initializing [ 7423.354376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7423.371312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7423.377394] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7424.777717] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7427.894829] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7429.116908] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7431.446703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7431.474504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7431.594598] Lustre: lustre-OST0000: new disk, initializing [ 7431.597559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7431.629691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7431.709874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7431.713245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7431.763475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7433.627749] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7436.818820] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7438.063211] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing lnet_if_list [ 7439.794995] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7439.807515] LNet: Added LNI 192.168.202.103@tcp42 [8/256/0/180] [ 7447.011058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7447.011304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7447.017286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7447.018368] LustreError: Skipped 1 previous similar message [ 7447.024176] Lustre: Skipped 1 previous similar message [ 7452.130968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7452.132576] Lustre: Skipped 1 previous similar message [ 7452.525856] LustreError: 479695:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7452.565820] Lustre: server umount lustre-OST0000 complete [ 7454.177212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7454.179906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7454.183086] Lustre: Skipped 1 previous similar message [ 7454.186038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7457.249406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7457.249796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7459.966699] LustreError: 479896:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7459.969297] LustreError: 479896:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7459.997561] Lustre: server umount lustre-MDT0000 complete [ 7461.304167] LustreError: 476337:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761970172 with bad export cookie 10390316721054611850 [ 7461.306466] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7461.307457] LustreError: 476337:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7461.373749] LustreError: 480097:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7461.376956] LustreError: 480097:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7461.430884] Lustre: server umount lustre-MDT0001 complete [ 7464.956263] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7466.168574] Key type lgssc unregistered [ 7466.340715] LNet: 480629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7466.344378] LNetError: 480629:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7466.346465] LNet: Removed LNI 192.168.202.103@tcp [ 7466.353440] LNet: Removed LNI 192.168.202.103@tcp42 [ 7466.651292] Key type .llcrypt unregistered [ 7466.652388] Key type ._llcrypt unregistered [ 7474.116481] Key type ._llcrypt registered [ 7474.117569] Key type .llcrypt registered [ 7474.146907] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7474.684890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7474.700833] alg: No test for adler32 (adler32-zlib) [ 7475.552857] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 7475.631786] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 7477.208170] Key type lgssc registered [ 7477.592289] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7480.221361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7480.227172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7481.385844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.705943] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7485.603352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7485.729567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7487.022050] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7488.864314] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7490.155374] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7492.569910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7492.705751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7494.644892] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7496.485248] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7515.774293] loop0: detected capacity change from 0 to 204800000 [ 7515.890550] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7515.893758] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7515.897200] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7515.923491] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7518.032622] loop0: detected capacity change from 0 to 204800000 [ 7518.141203] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7518.144515] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7518.150054] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7518.200884] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7519.948588] loop0: detected capacity change from 0 to 204800000 [ 7521.916573] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7521.935480] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7521.983340] Lustre: fs15246-OST0000: new disk, initializing [ 7521.985501] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7522.006089] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7524.034823] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7528.116107] loop1: detected capacity change from 0 to 204800000 [ 7530.104900] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7530.125834] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7530.165504] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7530.190058] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7530.227933] Lustre: fs15246-MDT0000: new disk, initializing [ 7530.278344] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7530.287625] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7530.326687] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7531.361095] Lustre: 486206:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000000621b4a0 x1847559649788928/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1761970258 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7531.698548] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7536.418104] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7536.421059] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7536.428872] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7540.710215] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7540.714753] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7545.723336] LustreError: 487321:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 7545.742453] Lustre: server umount fs15246-MDT0000 complete [ 7545.830927] LustreError: 481863:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7545.838314] LustreError: 481863:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7549.090102] LustreError: 487825:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 7549.092623] LustreError: 487825:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7549.110653] Lustre: server umount fs15246-OST0000 complete [ 7553.504821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7553.508875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7553.515651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7556.067696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7556.071373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7558.702997] LustreError: 488326:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7558.704865] LustreError: 488326:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7558.745876] Lustre: server umount lustre-OST0000 complete [ 7561.186582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7561.188085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7561.190386] Lustre: Skipped 1 previous similar message [ 7561.192204] Lustre: Skipped 1 previous similar message [ 7566.205528] LustreError: 488528:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7566.207788] LustreError: 488528:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7566.231634] Lustre: server umount lustre-MDT0000 complete [ 7566.305060] LustreError: 481839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7566.309290] LustreError: 481839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7567.482106] LustreError: 481819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761970278 with bad export cookie 14952905976555194806 [ 7567.484835] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7567.487632] LustreError: 481819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7567.627936] Lustre: server umount lustre-MDT0001 complete [ 7571.318267] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7572.391785] Key type lgssc unregistered [ 7572.520343] LNet: 489254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7572.524537] LNetError: 489254:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7572.534369] LNet: Removed LNI 192.168.202.103@tcp [ 7572.858369] Key type .llcrypt unregistered [ 7572.859836] Key type ._llcrypt unregistered [ 7580.601824] Key type ._llcrypt registered [ 7580.602764] Key type .llcrypt registered [ 7580.631330] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7580.987687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7581.043880] alg: No test for adler32 (adler32-zlib) [ 7581.902298] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 7581.994225] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 7583.576181] Key type lgssc registered [ 7583.979396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7586.615740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7588.278956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7589.956560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7591.555848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7597.086999] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 7600.345540] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7603.725562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7605.852797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7607.824963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7610.015311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7612.326647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7612.348993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7612.353826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7613.449587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7613.461382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7613.499074] Lustre: lustre-MDT0000: new disk, initializing [ 7613.528122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7613.534465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7614.889635] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7619.186728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7619.208260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7619.246303] Lustre: 493634:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7619.275643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7619.278279] Lustre: Skipped 1 previous similar message [ 7619.317680] Lustre: lustre-MDT0001: new disk, initializing [ 7619.358516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7619.375106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7619.380608] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7620.682301] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7623.868984] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7625.048913] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7627.434499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7627.459060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7627.561735] Lustre: lustre-OST0000: new disk, initializing [ 7627.563687] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7627.595345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7629.593737] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7630.971533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7630.975317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7631.007840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7632.885063] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7634.340681] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7634.413336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7635.734049] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7635.817715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7638.576097] LustreError: 496529:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7638.616026] Lustre: server umount lustre-OST0000 complete [ 7640.032828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7640.034890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7640.038668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7641.569511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7641.570093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7646.689414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7646.692498] Lustre: Skipped 2 previous similar messages [ 7651.809699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7651.812287] Lustre: Skipped 1 previous similar message [ 7654.368198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7654.426432] LustreError: 496730:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7654.429612] LustreError: 496730:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7654.479051] Lustre: server umount lustre-MDT0000 complete [ 7656.177146] LustreError: 494407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761970366 with bad export cookie 8737929975749467045 [ 7656.184125] LustreError: 494407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7656.184447] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7665.382696] LustreError: 496935:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7665.386598] LustreError: 496935:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7665.463420] Lustre: server umount lustre-MDT0001 complete [ 7676.571064] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 7681.345843] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7686.159548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7689.647405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7692.836850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7696.312633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7697.204263] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7576 sec ========== 00:13:28 (1761970408) [ 7698.110226] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:13:29 (1761970409) === [ 7699.750396] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:13:31 (1761970411) === [ 7718.776503] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7720.320615] Key type lgssc unregistered [ 7720.505767] LNet: 501762:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7720.510528] LNetError: 501762:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7720.520708] LNet: Removed LNI 192.168.202.103@tcp [ 7721.009906] Key type .llcrypt unregistered [ 7721.011996] Key type ._llcrypt unregistered