[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 538133772 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] BRK [0x0ec01000, 0x0ec01fff] PGTABLE [ 0.000000] BRK [0x0ec02000, 0x0ec02fff] PGTABLE [ 0.000000] BRK [0x0ec03000, 0x0ec03fff] PGTABLE [ 0.000000] BRK [0x0ec04000, 0x0ec04fff] PGTABLE [ 0.000000] BRK [0x0ec05000, 0x0ec05fff] PGTABLE [ 0.000000] BRK [0x0ec06000, 0x0ec06fff] PGTABLE [ 0.000000] BRK [0x0ec07000, 0x0ec07fff] PGTABLE [ 0.000000] BRK [0x0ec08000, 0x0ec08fff] PGTABLE [ 0.000000] BRK [0x0ec09000, 0x0ec09fff] PGTABLE [ 0.000000] BRK [0x0ec0a000, 0x0ec0afff] PGTABLE [ 0.000000] BRK [0x0ec0b000, 0x0ec0bfff] PGTABLE [ 0.000000] BRK [0x0ec0c000, 0x0ec0cfff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] On node 0 totalpages: 1076588 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 158 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:63 [ 0.000000] Normal zone: 4536 pages used for memmap [ 0.000000] Normal zone: 290304 pages, LIFO batch:63 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] pcpu-alloc: s221184 r8192 d28672 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001000] hpet clockevent registered [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002386] x2apic enabled [ 0.003000] Switched APIC routing to physical x2apic. [ 0.003014] kvm-guest: setup PV IPIs [ 0.006495] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009127] LSM: Security Framework initializing [ 0.010047] Yama: becoming mindful. [ 0.011036] SELinux: Initializing. [ 0.013070] *** VALIDATE selinux *** [ 0.022516] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027496] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030129] *** VALIDATE tmpfs *** [ 0.032215] *** VALIDATE proc *** [ 0.033283] *** VALIDATE cgroup *** [ 0.034020] *** VALIDATE cgroup2 *** [ 0.035000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035168] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037045] Spectre V2 : User space: Vulnerable [ 0.038013] Speculative Store Bypass: Vulnerable [ 0.042579] debug: unmapping init [mem 0xffffffffaba59000-0xffffffffaba60fff] [ 0.045373] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047215] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048047] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050010] ... generic registers: 4 [ 0.051010] ... value mask: 0000ffffffffffff [ 0.052019] ... max period: 00007fffffffffff [ 0.053029] ... fixed-purpose events: 3 [ 0.054029] ... event mask: 000000070000000f [ 0.055521] rcu: Hierarchical SRCU implementation. [ 0.057537] smp: Bringing up secondary CPUs ... [ 0.058616] x86: Booting SMP configuration: [ 0.059040] .... node #0, CPUs: #1 #2 #3 [ 0.080017] smp: Brought up 1 node, 4 CPUs [ 0.082014] smpboot: Max logical packages: 1 [ 0.083013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.123014] node 0 deferred pages initialised in 34ms [ 0.129163] devtmpfs: initialized [ 0.132389] x86/mm: Memory block size: 128MB [ 0.136235] gcov: version magic: 0x41383552 [ 0.139378] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141198] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147339] pinctrl core: initialized pinctrl subsystem [ 0.151949] ************************************************************* [ 0.156019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159018] ** ** [ 0.161016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163018] ** ** [ 0.166020] ** This means that this kernel is built to expose internal ** [ 0.169019] ** IOMMU data structures, which may compromise security on ** [ 0.171018] ** your system. ** [ 0.173023] ** ** [ 0.176023] ** If you see this message and you are not debugging the ** [ 0.177000] ** kernel, report this immediately to your vendor! ** [ 0.178014] ** ** [ 0.180011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182012] ************************************************************* [ 0.185995] NET: Registered protocol family 16 [ 0.187666] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.189079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.192090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.197445] cpuidle: using governor menu [ 0.199169] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202782] PCI: Using configuration type 1 for base access [ 0.205155] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217119] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225059] cryptd: max_cpu_qlen set to 1000 [ 0.227352] ACPI: Added _OSI(Module Device) [ 0.229014] ACPI: Added _OSI(Processor Device) [ 0.230024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.235802] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.242009] ACPI: Interpreter enabled [ 0.243065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.244015] ACPI: Using IOAPIC for interrupt routing [ 0.247181] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.248509] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.256754] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.258050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.261024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264095] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.269311] acpiphp: Slot [2] registered [ 0.271162] acpiphp: Slot [3] registered [ 0.272095] acpiphp: Slot [4] registered [ 0.274107] acpiphp: Slot [5] registered [ 0.275176] acpiphp: Slot [6] registered [ 0.276116] acpiphp: Slot [7] registered [ 0.278146] acpiphp: Slot [8] registered [ 0.279133] acpiphp: Slot [9] registered [ 0.282198] acpiphp: Slot [10] registered [ 0.284960] acpiphp: Slot [11] registered [ 0.286110] acpiphp: Slot [12] registered [ 0.287119] acpiphp: Slot [13] registered [ 0.288155] acpiphp: Slot [14] registered [ 0.290121] acpiphp: Slot [15] registered [ 0.293170] acpiphp: Slot [16] registered [ 0.296128] acpiphp: Slot [17] registered [ 0.298081] acpiphp: Slot [18] registered [ 0.299073] acpiphp: Slot [19] registered [ 0.300096] acpiphp: Slot [20] registered [ 0.301000] acpiphp: Slot [21] registered [ 0.301000] acpiphp: Slot [22] registered [ 0.302118] acpiphp: Slot [23] registered [ 0.303077] acpiphp: Slot [24] registered [ 0.304072] acpiphp: Slot [25] registered [ 0.305084] acpiphp: Slot [26] registered [ 0.306000] acpiphp: Slot [27] registered [ 0.307031] acpiphp: Slot [28] registered [ 0.308631] acpiphp: Slot [29] registered [ 0.310101] acpiphp: Slot [30] registered [ 0.311084] acpiphp: Slot [31] registered [ 0.312053] PCI host bridge to bus 0000:00 [ 0.313018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.315019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.318642] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.320034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.322023] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.323128] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.324000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.326000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.329846] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.338665] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.343599] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.346317] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.348024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.352025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.353000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.355674] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.359050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.365700] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.371015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.393023] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.397977] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.404283] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.412019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.420020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.441026] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.451940] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.458017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.463020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.478036] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.491886] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.500016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.502022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.517029] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.526502] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.532022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.538022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.549016] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.555076] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.559017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.567048] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.583000] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.591449] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.608019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.618017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.636016] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.653650] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.655311] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.661358] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.665333] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.670227] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.676109] iommu: Default domain type: Passthrough [ 0.677000] SCSI subsystem initialized [ 0.677000] ACPI: bus type USB registered [ 0.678099] usbcore: registered new interface driver usbfs [ 0.683225] usbcore: registered new interface driver hub [ 0.688305] usbcore: registered new device driver usb [ 0.689233] pps_core: LinuxPPS API ver. 1 registered [ 0.691030] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.696154] PTP clock support registered [ 0.701047] EDAC MC: Ver: 3.0.0 [ 0.702094] PCI: Using ACPI for IRQ routing [ 0.705018] PCI: pci_cache_line_size set to 64 bytes [ 0.705729] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.705743] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 0.705755] e820: reserve RAM buffer [mem 0x146e00000-0x147ffffff] [ 0.706792] NetLabel: Initializing [ 0.709015] NetLabel: domain hash size = 128 [ 0.711011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.714141] NetLabel: unlabeled traffic allowed by default [ 0.717454] vgaarb: loaded [ 0.719341] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.724022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.737711] clocksource: Switched to clocksource kvm-clock [ 0.903718] VFS: Disk quotas dquot_6.6.0 [ 0.905396] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.908985] *** VALIDATE ramfs *** [ 0.910336] *** VALIDATE hugetlbfs *** [ 0.911886] pnp: PnP ACPI init [ 0.913861] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.913957] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.913986] pnp 00:02: [dma 2] [ 0.914057] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.914151] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.914213] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.914283] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.914829] pnp: PnP ACPI: found 6 devices [ 0.932199] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.935399] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.937510] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.939717] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.941893] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.944450] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.947146] NET: Registered protocol family 2 [ 0.949409] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.954115] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.957224] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.962054] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.964986] TCP: Hash tables configured (established 65536 bind 65536) [ 0.967546] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.970203] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.972627] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.975309] NET: Registered protocol family 1 [ 0.978788] RPC: Registered named UNIX socket transport module. [ 0.980689] RPC: Registered udp transport module. [ 0.982395] RPC: Registered tcp transport module. [ 0.983833] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.985923] NET: Registered protocol family 44 [ 0.988203] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.990102] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.991912] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.995223] PCI: CLS 0 bytes, default 64 [ 0.997154] Unpacking initramfs... [ 2.662685] debug: unmapping init [mem 0xffff8ffb3cc54000-0xffff8ffb3ffbffff] [ 2.667572] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.669645] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.672377] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.203830] Initialise system trusted keyrings [ 3.205268] Key type blacklist registered [ 3.207209] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.216541] zbud: loaded [ 3.219196] *** VALIDATE nfs *** [ 3.220418] *** VALIDATE nfs4 *** [ 3.221600] pstore: using deflate compression [ 3.227718] Platform Keyring initialized [ 3.343525] NET: Registered protocol family 38 [ 3.345349] Key type asymmetric registered [ 3.346890] Asymmetric key parser 'x509' registered [ 3.348979] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.353161] io scheduler mq-deadline registered [ 3.354875] io scheduler kyber registered [ 3.356504] io scheduler bfq registered [ 3.358308] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.366672] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.370470] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.373409] ACPI: Power Button [PWRF] [ 3.481347] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.585395] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.797602] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.892406] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.078331] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.108828] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.139128] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.147647] Non-volatile memory driver v1.3 [ 4.150241] Linux agpgart interface v0.103 [ 4.218774] virtio_blk virtio1: [vda] 132768 512-byte logical blocks (68.0 MB/64.8 MiB) [ 4.222364] vda: detected capacity change from 0 to 67977216 [ 4.246778] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.252816] vdb: detected capacity change from 0 to 1073741824 [ 4.274668] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.277715] vdc: detected capacity change from 0 to 2621440000 [ 4.296846] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.299886] vdd: detected capacity change from 0 to 2621440000 [ 4.316260] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.318993] vde: detected capacity change from 0 to 4294967296 [ 4.334796] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.337878] vdf: detected capacity change from 0 to 4294967296 [ 4.343381] libphy: Fixed MDIO Bus: probed [ 4.351488] usbcore: registered new interface driver usbserial_generic [ 4.353899] usbserial: USB Serial support registered for generic [ 4.356276] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.360665] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.363295] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.366708] mousedev: PS/2 mouse device common for all mice [ 4.372641] rtc_cmos 00:05: RTC can wake from S4 [ 4.375931] rtc_cmos 00:05: registered as rtc0 [ 4.379192] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.384646] intel_pstate: CPU model not supported [ 4.392195] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.405104] hid: raw HID events driver (C) Jiri Kosina [ 4.407503] usbcore: registered new interface driver usbhid [ 4.409352] usbhid: USB HID core driver [ 4.410825] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.412254] drop_monitor: Initializing network drop monitor service [ 4.412374] Initializing XFRM netlink socket [ 4.412737] NET: Registered protocol family 10 [ 4.415834] Segment Routing with IPv6 [ 4.429350] NET: Registered protocol family 17 [ 4.432887] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.436266] mpls_gso: MPLS GSO support [ 4.437650] start plist test [ 4.441840] end plist test [ 4.444946] RAS: Correctable Errors collector initialized. [ 4.448629] AVX version of gcm_enc/dec engaged. [ 4.451167] AES CTR mode by8 optimization enabled [ 4.584491] sched_clock: Marking stable (4584007470, 0)->(5748156074, -1164148604) [ 4.588163] registered taskstats version 1 [ 4.591873] Loading compiled-in X.509 certificates [ 4.599516] zswap: loaded using pool lzo/zbud [ 4.642666] Key type big_key registered [ 4.662031] Key type encrypted registered [ 4.663282] ima: No TPM chip found, activating TPM-bypass! [ 4.664832] ima: Allocated hash algorithm: sha1 [ 4.669936] ima: No architecture policies found [ 4.671729] evm: Initialising EVM extended attributes: [ 4.673285] evm: security.selinux [ 4.674352] evm: security.ima [ 4.675288] evm: security.capability [ 4.676369] evm: HMAC attrs: 0x1 [ 4.679190] rtc_cmos 00:05: setting system clock to 2025-08-01 00:46:16 UTC (1754009176) [ 4.690185] debug: unmapping init [mem 0xffffffffaca03000-0xffffffffacbfffff] [ 4.695953] debug: unmapping init [mem 0xffffffffab782000-0xffffffffaba58fff] [ 4.702193] Write protecting the kernel read-only data: 28672k [ 4.706804] debug: unmapping init [mem 0xffffffffa9e03000-0xffffffffa9ffffff] [ 4.709780] debug: unmapping init [mem 0xffffffffaa714000-0xffffffffaa7fffff] [ 4.744868] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.753152] systemd[1]: Detected virtualization kvm. [ 4.754962] systemd[1]: Detected architecture x86-64. [ 4.756706] systemd[1]: Running in initial RAM disk. [ 4.791180] systemd[1]: No hostname configured. [ 4.793230] systemd[1]: Set hostname to . [ 4.795302] random: systemd: uninitialized urandom read (16 bytes read) [ 4.797577] systemd[1]: Initializing machine ID from random generator. [ 5.001214] random: systemd: uninitialized urandom read (16 bytes read) [ 5.003431] systemd[1]: Reached target Initrd Root Device. [ 5.007555] random: systemd: uninitialized urandom read (16 bytes read) [ 5.009984] systemd[1]: Listening on udev Kernel Socket. [ 5.021645] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ 6.058277] device-mapper: uevent: version 1.0.3 [ 6.060662] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ 7.234985] virtio_net virtio0 ens2: renamed from eth0 [ 7.259770] random: fast init done [ 7.672539] libata version 3.00 loaded. [ 8.285558] ata_piix 0000:00:01.1: version 2.13 [ 8.390653] scsi host0: ata_piix [ 8.440278] scsi host1: ata_piix [ 8.441618] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.444023] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.192890] random: crng init done [ 12.194170] random: 7 urandom warning(s) missed due to ratelimiting [ 15.948961] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 17.845855] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.248121] SELinux: Disabled at runtime. [ 18.333424] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.348440] systemd[1]: Detected virtualization kvm. [ 18.351410] systemd[1]: Detected architecture x86-64. [ 19.289841] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.292530] systemd[1]: Stopped Switch Root. [ 19.296513] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.307066] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.313373] systemd[1]: Stopped Journal Service. [ 19.322618] systemd[1]: Starting Journal Service... [ 19.347037] systemd[1]: Listening on RPCbind Server Activation Socket. [ 19.504621] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 20.459307] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 21.379685] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.471656] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.618878] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.660040] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 22.660063] EDAC sbridge: Ver: 1.1.2 [ 25.202585] Key type dns_resolver registered [ 25.577340] NFS: Registering the id_resolver key type [ 25.579471] Key type id_resolver registered [ 25.582094] Key type id_legacy registered [ 53.683403] libcfs: loading out-of-tree module taints kernel. [ 53.694505] Key type ._llcrypt registered [ 53.696054] Key type .llcrypt registered [ 53.747616] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 61.950777] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 62.632579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.641046] alg: No test for adler32 (adler32-zlib) [ 63.687087] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 64.045189] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 64.048425] LNet: Accept secure, port 988 [ 65.688565] Key type lgssc registered [ 66.270286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 73.951646] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 75.038971] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.020544] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 81.999434] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 85.285426] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 90.970065] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 96.050602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 96.075512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 96.096664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 97.214420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 97.230723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 97.275645] Lustre: lustre-MDT0000: new disk, initializing [ 97.318951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 97.330461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 98.939877] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 104.971670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 105.006353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 105.036597] Lustre: 6450:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 105.052292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 105.054667] Lustre: Skipped 1 previous similar message [ 105.099393] Lustre: lustre-MDT0001: new disk, initializing [ 105.123687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 105.135831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 105.140601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 106.632975] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 109.226605] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 113.247124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 113.279805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 113.385473] Lustre: lustre-OST0000: new disk, initializing [ 113.388490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 113.415481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 115.723246] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 121.492318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 121.515230] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 121.546908] Lustre: lustre-OST0001: new disk, initializing [ 121.548695] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 121.564311] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 121.873065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 121.876042] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 121.904630] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 121.906345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 123.494070] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 129.916455] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 136.892666] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 144.075957] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing check_logdir /tmp/testlogs/ [ 145.585993] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing yml_node [ 147.307213] Lustre: DEBUG MARKER: Client: 2.16.57.19 [ 148.234891] Lustre: DEBUG MARKER: MDS: 2.16.57.19 [ 149.257361] Lustre: DEBUG MARKER: OSS: 2.16.57.19 [ 149.894468] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jul 31 20:48:41 EDT 2025 [ 156.299151] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 156.833527] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 172.000585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 172.003067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 172.008296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 173.025183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 173.025724] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 173.029331] Lustre: Skipped 2 previous similar messages [ 173.034422] Lustre: Skipped 2 previous similar messages [ 174.720035] LustreError: 12111:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 174.760654] Lustre: server umount lustre-MDT0000 complete [ 176.234743] LustreError: 10190:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754009348 with bad export cookie 11857215646088106379 [ 176.235814] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 176.239064] LustreError: 10190:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 178.144622] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 178.145139] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 178.150138] Lustre: Skipped 1 previous similar message [ 178.152342] Lustre: Skipped 1 previous similar message [ 181.152732] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 181.155134] Lustre: Skipped 2 previous similar messages [ 182.336717] LustreError: 12313:0:(obd_class.h:479:obd_check_dev()) Device 26 not setup [ 182.339435] LustreError: 12313:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 182.440417] Lustre: server umount lustre-MDT0001 complete [ 184.007195] LustreError: 12514:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 184.009593] LustreError: 12514:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 184.031556] Lustre: server umount lustre-OST0000 complete [ 185.570766] Lustre: server umount lustre-OST0001 complete [ 187.815957] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 191.491351] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 195.572982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 198.108214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 200.552888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 203.152967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 206.199416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 206.230155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 206.330276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 206.349055] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 206.389204] Lustre: lustre-MDT0000: new disk, initializing [ 206.431049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 206.439619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 206.442937] Lustre: Skipped 1 previous similar message [ 208.068557] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 213.233360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 213.266665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 213.312766] Lustre: 14877:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 213.454764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 213.460367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 213.466396] Lustre: Skipped 1 previous similar message [ 215.064798] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 218.703141] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 220.435646] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 223.452416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 223.490656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 223.606752] Lustre: lustre-OST0000: new disk, initializing [ 223.608808] Lustre: Skipped 1 previous similar message [ 223.611571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 223.613944] Lustre: Skipped 2 previous similar messages [ 223.651242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.654969] Lustre: Skipped 1 previous similar message [ 225.287678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 225.291237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 225.340578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 225.991900] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 229.798078] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 231.588267] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 231.691854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 233.656436] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 233.780460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 235.492320] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 235.493170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 235.496439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 235.497232] LustreError: Skipped 1 previous similar message [ 235.501729] Lustre: Skipped 1 previous similar message [ 240.749154] LustreError: 17782:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 240.756095] LustreError: 17782:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 240.838471] Lustre: server umount lustre-OST0000 complete [ 244.193166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 244.196687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 244.202432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 244.205238] Lustre: Skipped 3 previous similar messages [ 248.658256] LustreError: 17983:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 248.661015] LustreError: 17983:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 248.715032] Lustre: server umount lustre-MDT0000 complete [ 250.559980] LustreError: 17985:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754009422 with bad export cookie 11857215646088107814 [ 250.563373] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 250.563787] LustreError: 17985:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 250.740196] Lustre: server umount lustre-MDT0001 complete [ 252.711602] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 20:50:23 (1754009423) [ 253.844602] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 254.663610] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 20:50:25 (1754009425) [ 258.070622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 258.289684] LustreError: 18855:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 258.299254] LustreError: 18855:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 258.340349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 260.241269] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 263.649830] LustreError: 18855:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 264.500862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 266.464780] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 268.920470] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 270.514860] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 273.388375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 275.822962] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 278.052536] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 294.372295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.372316] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 294.377373] Lustre: Skipped 1 previous similar message [ 294.382178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 294.382185] Lustre: Skipped 2 previous similar messages [ 297.452613] LustreError: 21103:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 297.455748] LustreError: 21103:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 297.524743] Lustre: server umount lustre-OST0000 complete [ 313.312146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 313.370593] Lustre: server umount lustre-MDT0000 complete [ 314.795653] LustreError: 21306:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754009486 with bad export cookie 11857215646088108696 [ 314.798346] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 314.800528] LustreError: 21306:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 314.868214] LustreError: 21508:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 314.870664] LustreError: 21508:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 314.948757] Lustre: server umount lustre-MDT0001 complete [ 319.980264] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 321.142205] Key type lgssc unregistered [ 321.293412] LNet: 22034:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.342392] LNet: Removed LNI 192.168.203.124@tcp [ 322.724150] Key type .llcrypt unregistered [ 322.725484] Key type ._llcrypt unregistered [ 332.243524] Key type ._llcrypt registered [ 332.245167] Key type .llcrypt registered [ 332.289213] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 332.773852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 332.788384] alg: No test for adler32 (adler32-zlib) [ 333.652046] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 333.747638] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 333.749607] LNet: Accept secure, port 988 [ 335.336153] Key type lgssc registered [ 335.722772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 338.807352] loop: module loaded [ 338.829488] loop0: detected capacity change from 0 to 204800000 [ 338.953675] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 338.958282] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 339.086903] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 339.091660] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 339.119472] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 339.147370] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 339.164982] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 339.197506] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 341.387584] loop0: detected capacity change from 0 to 204800000 [ 341.459156] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 341.464561] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 341.469979] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 341.530586] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 343.465941] loop0: detected capacity change from 0 to 204800000 [ 345.645970] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 345.668173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 345.672839] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 346.765486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 346.783569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 346.823458] Lustre: lustre-MDT0000: new disk, initializing [ 346.850730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 346.858780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 348.313801] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 352.171578] loop1: detected capacity change from 0 to 204800000 [ 354.282571] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 354.307796] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 354.421340] Lustre: lustre-OST07c6: new disk, initializing [ 354.425124] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 354.428359] Lustre: Skipped 1 previous similar message [ 354.455330] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 356.555375] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 363.009977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 363.013556] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 363.025320] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 450.017524] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 450.021066] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 450.027474] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 455.137728] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 460.257260] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 461.280105] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 461.296087] LustreError: 26379:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 461.359434] Lustre: server umount lustre-OST07c6 complete [ 465.028739] LustreError: 26880:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 465.031622] LustreError: 26880:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 465.123882] Lustre: server umount lustre-MDT0000 complete [ 472.400234] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 473.575910] Key type lgssc unregistered [ 473.738534] LNet: 27708:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.794628] LNet: Removed LNI 192.168.203.124@tcp [ 475.147787] Key type .llcrypt unregistered [ 475.149510] Key type ._llcrypt unregistered [ 482.666707] Key type ._llcrypt registered [ 482.667942] Key type .llcrypt registered [ 482.705532] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 483.050398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 483.107436] alg: No test for adler32 (adler32-zlib) [ 483.978625] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 484.064158] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 484.066206] LNet: Accept secure, port 988 [ 485.648274] Key type lgssc registered [ 486.026231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 488.844146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 488.851974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.031365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.396639] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 494.332880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 494.463263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 495.847246] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 497.789363] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 499.168461] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 501.618833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 501.757556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 503.867516] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 505.826068] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 515.107758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 518.363979] LustreError: 31122:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 523.747197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 523.747242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.750060] LustreError: Skipped 1 previous similar message [ 523.755442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 525.282021] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 525.282641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 525.284357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.022481] LustreError: 31373:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 527.078629] Lustre: server umount lustre-OST0000 complete [ 530.401608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 530.401874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.404800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 530.407607] Lustre: Skipped 1 previous similar message [ 530.409953] Lustre: Skipped 2 previous similar messages [ 534.461369] LustreError: 31574:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 534.463386] LustreError: 31574:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 534.489860] Lustre: server umount lustre-MDT0000 complete [ 535.520947] LustreError: 28924:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.524788] LustreError: 28924:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 535.766627] LustreError: 28906:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754009707 with bad export cookie 13123930047082435914 [ 535.769585] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 535.770327] LustreError: 28906:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 535.825332] LustreError: 31776:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 535.826976] LustreError: 31776:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 535.908460] Lustre: server umount lustre-MDT0001 complete [ 540.574195] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 541.698282] Key type lgssc unregistered [ 541.837439] LNet: 32303:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 542.886468] LNet: Removed LNI 192.168.203.124@tcp [ 543.202060] Key type .llcrypt unregistered [ 543.203528] Key type ._llcrypt unregistered [ 550.439023] Key type ._llcrypt registered [ 550.439833] Key type .llcrypt registered [ 550.474497] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 550.828251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 550.875809] alg: No test for adler32 (adler32-zlib) [ 551.738316] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 551.826399] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 551.828580] LNet: Accept secure, port 988 [ 553.408147] Key type lgssc registered [ 553.768268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 556.630887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 556.637492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.824440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 559.204438] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 562.354082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 562.520763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 564.130661] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 566.100741] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 567.383455] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 569.808238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 569.931557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.857141] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 573.790325] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 582.248439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 588.768925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 588.771556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.776272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 592.868180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 592.869564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.870053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 594.606141] LustreError: 35769:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 594.667299] Lustre: server umount lustre-OST0000 complete [ 597.985631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.986448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 597.990872] Lustre: Skipped 1 previous similar message [ 597.992637] Lustre: Skipped 1 previous similar message [ 603.105134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 603.107302] Lustre: Skipped 1 previous similar message [ 608.225503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 608.228116] Lustre: Skipped 1 previous similar message [ 610.272127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 610.305547] LustreError: 35971:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 610.308256] LustreError: 35971:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 610.337839] Lustre: server umount lustre-MDT0000 complete [ 611.669570] LustreError: 33500:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754009783 with bad export cookie 9928491883585074934 [ 611.672970] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 611.674399] LustreError: 33500:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 611.741233] LustreError: 36173:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 611.743430] LustreError: 36173:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 611.824747] Lustre: server umount lustre-MDT0001 complete [ 616.784940] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 617.988937] Key type lgssc unregistered [ 618.137427] LNet: 36700:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 619.174370] LNet: Removed LNI 192.168.203.124@tcp [ 619.519119] Key type .llcrypt unregistered [ 619.520688] Key type ._llcrypt unregistered [ 628.476096] Key type ._llcrypt registered [ 628.477113] Key type .llcrypt registered [ 628.513828] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 628.906567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 628.929560] alg: No test for adler32 (adler32-zlib) [ 629.797958] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 629.885030] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 629.886890] LNet: Accept secure, port 988 [ 631.473082] Key type lgssc registered [ 631.862444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 634.651520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 634.660463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.833243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 637.210929] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 640.188568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 640.336848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 641.673624] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 643.504287] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 644.714840] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 646.951318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 647.063591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 648.947753] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 650.749268] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 660.391391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 665.571482] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 665.572028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 665.576123] Lustre: Skipped 1 previous similar message [ 666.479385] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 666.483056] Lustre: Skipped 2 previous similar messages [ 669.693505] LustreError: 40315:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 669.724806] Lustre: server umount lustre-MDT0000 complete [ 670.690581] LustreError: 38771:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 670.695976] LustreError: 38771:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 671.061502] LustreError: 38046:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754009842 with bad export cookie 8182845792531263994 [ 671.063772] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 671.067021] LustreError: 38046:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 671.588877] LustreError: 38060:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 671.597162] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.24@tcp (stopping) [ 675.808885] LustreError: 38061:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 675.809206] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 675.819741] Lustre: Skipped 1 previous similar message [ 675.822348] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 677.188282] LustreError: 40518:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 677.190769] LustreError: 40518:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 677.271645] Lustre: server umount lustre-MDT0001 complete [ 695.213902] LustreError: 40741:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 695.215612] LustreError: 40741:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 695.278724] Lustre: server umount lustre-OST0000 complete [ 701.986884] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 703.150324] Key type lgssc unregistered [ 703.296400] LNet: 41573:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.358350] LNet: Removed LNI 192.168.203.124@tcp [ 704.679511] Key type .llcrypt unregistered [ 704.680348] Key type ._llcrypt unregistered [ 713.802103] Key type ._llcrypt registered [ 713.803483] Key type .llcrypt registered [ 713.843263] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 714.211737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 714.240447] alg: No test for adler32 (adler32-zlib) [ 715.090421] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 715.175308] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 715.177649] LNet: Accept secure, port 988 [ 716.760165] Key type lgssc registered [ 717.108962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 719.729718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 719.741996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 720.905153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 722.203891] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 724.944801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 725.063285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 726.371441] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 728.139236] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 729.410541] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 731.731385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 731.852926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 733.680168] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 735.364931] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 745.190603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 750.563573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 750.563830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 750.564335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 750.567828] Lustre: Skipped 1 previous similar message [ 755.682087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 755.684719] Lustre: Skipped 3 previous similar messages [ 760.802036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 760.804248] Lustre: Skipped 2 previous similar messages [ 762.336091] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 762.351182] LustreError: 45184:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 762.407062] Lustre: server umount lustre-OST0000 complete [ 775.145837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.151339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 775.153741] Lustre: Skipped 1 previous similar message [ 780.257024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 780.259532] Lustre: Skipped 2 previous similar messages [ 781.371965] LustreError: 45560:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 781.373405] LustreError: 45560:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 781.398374] Lustre: server umount lustre-MDT0000 complete [ 782.602211] LustreError: 42915:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754009954 with bad export cookie 1426710394249332464 [ 782.604709] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 782.605660] LustreError: 42915:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 788.732070] LustreError: 45761:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 788.733894] LustreError: 45761:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 788.804117] Lustre: server umount lustre-MDT0001 complete [ 793.401077] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 794.549677] Key type lgssc unregistered [ 794.686366] LNet: 46287:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.750365] LNet: Removed LNI 192.168.203.124@tcp [ 796.055987] Key type .llcrypt unregistered [ 796.057052] Key type ._llcrypt unregistered [ 804.954673] Key type ._llcrypt registered [ 804.956073] Key type .llcrypt registered [ 805.000937] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 805.488602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 805.501256] alg: No test for adler32 (adler32-zlib) [ 806.355846] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 806.441637] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 806.443568] LNet: Accept secure, port 988 [ 808.025392] Key type lgssc registered [ 808.427279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 811.231891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 811.239540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.396342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 813.650356] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 816.462496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 816.577691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 817.818952] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 819.532384] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 820.674651] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 822.902031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 823.012696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 824.810130] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 826.499479] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 837.350434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 844.179806] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 845.331303] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 853.476199] LNetError: Refusing connection from 192.168.203.124 for 127.0.0.2@tcp: No matching NI [ 853.478740] LNetError: 46955:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 853.481987] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 856.053460] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-07-3121h00m27s [ 856.520640] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 857.449038] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 857.568772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 857.570923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 857.575776] LNetError: Refusing connection from 192.168.203.124 for 127.0.0.2@tcp: No matching NI [ 857.578729] LNetError: 46958:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 857.582026] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 858.467830] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 858.469865] Lustre: Skipped 3 previous similar messages [ 863.356988] LustreError: 50216:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 863.386572] Lustre: server umount lustre-MDT0000 complete [ 863.712747] LNetError: Refusing connection from 192.168.203.124 for 127.0.0.2@tcp: No matching NI [ 863.716247] LNetError: 46956:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 863.719072] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 864.451808] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 864.611742] LustreError: 47647:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 864.616577] LustreError: 47647:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 866.537263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 866.577385] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 866.657503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.684082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 867.685146] LNetError: Refusing connection from 192.168.203.124 for 127.0.0.2@tcp: No matching NI [ 867.687039] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 867.688323] LNetError: 46957:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 867.692620] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 867.924156] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 869.067107] Lustre: DEBUG MARKER: Wait for df (35961) ... [ 869.592733] Lustre: DEBUG MARKER: done [ 871.905149] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 871.908548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.913771] Lustre: Skipped 2 previous similar messages [ 871.915616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 871.917542] Lustre: Skipped 3 previous similar messages [ 872.931094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 876.587575] LustreError: 51257:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 876.589268] LustreError: 51257:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 876.643320] Lustre: server umount lustre-OST0000 complete [ 878.049795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 878.049884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 878.053764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 878.056142] Lustre: Skipped 1 previous similar message [ 878.057687] LNetError: Refusing connection from 192.168.203.124 for 127.0.0.2@tcp: No matching NI [ 878.057946] Lustre: Skipped 3 previous similar messages [ 878.058989] LNetError: 46955:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 878.066028] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 884.129282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 884.132635] Lustre: Skipped 3 previous similar messages [ 892.384200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 892.415431] LustreError: 51462:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 892.418020] LustreError: 51462:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 892.445625] Lustre: server umount lustre-MDT0000 complete [ 893.409437] LustreError: 47651:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 893.416557] LustreError: 47651:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 893.746719] LustreError: 47633:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010065 with bad export cookie 1580263685443542339 [ 893.747866] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 893.751784] LustreError: 47633:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 893.900769] Lustre: server umount lustre-MDT0001 complete [ 898.563850] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 899.674464] Key type lgssc unregistered [ 899.815434] LNet: 52203:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.838705] LNet: Removed LNI 192.168.203.124@tcp [ 901.149580] Key type .llcrypt unregistered [ 901.151319] Key type ._llcrypt unregistered [ 912.159180] Key type ._llcrypt registered [ 912.160700] Key type .llcrypt registered [ 912.199615] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 918.315769] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 918.744421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 918.755423] alg: No test for adler32 (adler32-zlib) [ 919.610105] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 919.685781] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 919.687683] LNet: Accept secure, port 988 [ 921.264159] Key type lgssc registered [ 921.629776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 924.618412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 926.852837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 928.947149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 931.156331] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 933.444058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 933.461428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 933.466476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.552271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 934.564743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 934.603577] Lustre: lustre-MDT0000: new disk, initializing [ 934.633141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 934.640915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 935.951042] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 940.111573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 940.137170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 940.172878] Lustre: 56109:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 940.197237] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 940.199348] Lustre: Skipped 1 previous similar message [ 940.234996] Lustre: lustre-MDT0001: new disk, initializing [ 940.266811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 940.278221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 940.281668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 941.560436] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 944.646592] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 945.843101] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 948.083645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 948.107047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 948.201692] Lustre: lustre-OST0000: new disk, initializing [ 948.203979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 948.237169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 949.943594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 949.947641] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 949.979448] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 950.180977] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 953.315555] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 954.677121] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 954.759691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 956.132692] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 956.226260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 960.482235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 960.483298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 960.485106] LustreError: Skipped 1 previous similar message [ 960.485594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 960.488351] Lustre: Skipped 1 previous similar message [ 960.494166] Lustre: Skipped 1 previous similar message [ 965.602600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 965.604291] Lustre: Skipped 1 previous similar message [ 970.722897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 970.725079] Lustre: Skipped 1 previous similar message [ 971.232143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 971.251497] LustreError: 59020:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 971.309913] Lustre: server umount lustre-OST0000 complete [ 975.842134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.844416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 975.847475] Lustre: Skipped 1 previous similar message [ 978.813414] LustreError: 59228:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 978.816393] LustreError: 59228:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 978.852629] Lustre: server umount lustre-MDT0000 complete [ 980.181247] LustreError: 57846:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010152 with bad export cookie 9461055201704648156 [ 980.183450] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 980.185923] LustreError: 57846:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 980.250197] LustreError: 59430:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 980.252722] LustreError: 59430:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 980.331715] Lustre: server umount lustre-MDT0001 complete [ 983.143411] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 21:02:34 (1754010154) [ 985.550078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 985.723728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 987.057455] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 990.066806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 990.195220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 991.699640] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 993.737315] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 995.059356] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 997.501250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 999.686928] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1001.638407] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1014.407381] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-07-3121h03m05s [ 1014.942565] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1017.208650] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1018.064789] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1018.080375] LustreError: 60808:0:(mdt_handler.c:2329:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1022.948867] LNetError: Refusing connection from 192.168.203.124 for 127.0.0.2@tcp: No matching NI [ 1022.952371] LNetError: 54317:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1022.955497] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1038.194428] Lustre: lustre-MDT0000: Client 1d28b5d9-060b-4c00-903b-028de39c3922 (at 192.168.203.24@tcp) reconnecting [ 1058.120073] LustreError: 60808:0:(mdt_handler.c:2329:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1058.745346] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1059.468628] Lustre: DEBUG MARKER: done [ 1061.344737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1061.346857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.351365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1061.352837] Lustre: Skipped 1 previous similar message [ 1062.370249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1066.862240] LustreError: 63103:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1066.864931] LustreError: 63103:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1066.916698] Lustre: server umount lustre-OST0000 complete [ 1072.096791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1072.098652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1072.102204] Lustre: Skipped 1 previous similar message [ 1072.103639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1072.105366] Lustre: Skipped 2 previous similar messages [ 1072.107311] LNetError: Refusing connection from 192.168.203.124 for 127.0.0.2@tcp: No matching NI [ 1072.109115] LNetError: 54320:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1072.111096] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1074.235287] LustreError: 63305:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1074.236942] LustreError: 63305:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1074.258243] Lustre: server umount lustre-MDT0000 complete [ 1075.428198] LustreError: 60082:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010247 with bad export cookie 9461055201704649017 [ 1075.430504] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1075.433243] LustreError: 60082:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1075.564323] Lustre: server umount lustre-MDT0001 complete [ 1079.789315] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1080.784225] Key type lgssc unregistered [ 1080.920428] LNet: 64034:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.958336] LNet: Removed LNI 192.168.203.124@tcp [ 1082.233086] Key type .llcrypt unregistered [ 1082.233884] Key type ._llcrypt unregistered [ 1093.089143] Key type ._llcrypt registered [ 1093.090076] Key type .llcrypt registered [ 1093.125990] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 1098.739330] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1099.012224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1099.020070] alg: No test for adler32 (adler32-zlib) [ 1099.870241] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 1099.943917] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1099.945291] LNet: Accept secure, port 988 [ 1101.520132] Key type lgssc registered [ 1101.849525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1104.771136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1106.870075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1108.722069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1110.746125] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1113.022146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1113.039490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1113.046260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1114.129246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1114.139339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1114.179312] Lustre: lustre-MDT0000: new disk, initializing [ 1114.202857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1114.210198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1115.511599] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1119.624756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1119.645994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1119.680426] Lustre: 67939:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1119.706371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1119.708176] Lustre: Skipped 1 previous similar message [ 1119.744136] Lustre: lustre-MDT0001: new disk, initializing [ 1119.774825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1119.786713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1119.789503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1121.147233] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1124.228547] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1125.427970] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1127.740407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1127.764256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1127.859412] Lustre: lustre-OST0000: new disk, initializing [ 1127.861873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1127.891570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.295674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1128.300150] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1128.352119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1129.795022] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1132.745529] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1133.979361] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1134.052805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1135.254388] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1135.333336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1138.659304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1138.659721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1138.662110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1138.667489] Lustre: Skipped 1 previous similar message [ 1138.671946] Lustre: Skipped 1 previous similar message [ 1141.995107] LustreError: 70845:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1142.055676] Lustre: server umount lustre-OST0000 complete [ 1143.777161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1143.777888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1143.781200] Lustre: Skipped 1 previous similar message [ 1148.897592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1148.900695] Lustre: Skipped 2 previous similar messages [ 1149.375729] LustreError: 71046:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1149.378779] LustreError: 71046:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1149.412710] Lustre: server umount lustre-MDT0000 complete [ 1150.775180] LustreError: 67930:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010322 with bad export cookie 5570632088982721957 [ 1150.778522] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1150.779106] LustreError: 67930:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1150.843397] LustreError: 71247:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 1150.846133] LustreError: 71247:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1150.934504] Lustre: server umount lustre-MDT0001 complete [ 1153.792873] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 21:05:25 (1754010325) [ 1154.331623] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1154.914860] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 21:05:26 (1754010326) [ 1158.754967] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1160.158357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.311358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1160.671278] Lustre: Failing over lustre-MDT0000 [ 1160.673136] LustreError: 72312:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1160.677362] LustreError: 72312:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1160.679463] LustreError: 72312:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1160.738110] LustreError: 72363:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1160.739949] LustreError: 72363:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1160.810389] Lustre: server umount lustre-MDT0000 complete [ 1162.940810] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:05:34 (1754010334) [ 1165.220066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1165.357974] LustreError: 72929:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1165.365248] LustreError: 72929:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1165.373458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1166.645201] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1169.684297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1171.245877] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1173.069345] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1174.285663] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1176.579461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1176.687899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1176.690097] Lustre: Skipped 1 previous similar message [ 1178.551091] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1180.424958] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1197.885065] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1200.609179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1200.611836] LustreError: Skipped 1 previous similar message [ 1200.614060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.619872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1205.560561] LustreError: 75274:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1205.562786] LustreError: 75274:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1205.589450] Lustre: server umount lustre-MDT0000 complete [ 1206.753844] LustreError: 72925:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1206.760111] LustreError: 72925:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1206.777969] LustreError: 72910:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010378 with bad export cookie 5570632088982723042 [ 1206.782399] LustreError: 72910:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1206.783096] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1206.910881] Lustre: server umount lustre-MDT0001 complete [ 1208.029544] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1209.520952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1212.249082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1216.992647] LustreError: 66157:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c6125b31 x1839213073981056/t0(0) o250->MGC192.168.203.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1217.100985] LustreError: 74523:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1217.101316] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.113146] Lustre: Skipped 2 previous similar messages [ 1217.133651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1217.155389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1218.323945] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1221.049942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1222.362602] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1224.069860] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1225.228980] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1226.213645] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1227.232476] Lustre: 66161:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754010383/real 1754010383] req@000000004ca99af2 x1839213073980800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1754010399 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1231.337679] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1233.248134] Lustre: 66158:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754010388/real 1754010388] req@00000000666b0946 x1839213073980928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1754010404 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1236.450730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1236.451192] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.451954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1236.451959] Lustre: Skipped 3 previous similar messages [ 1236.458808] Lustre: Skipped 1 previous similar message [ 1238.330776] LustreError: 77723:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1238.332731] LustreError: 77723:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1238.360480] Lustre: server umount lustre-MDT0000 complete [ 1239.556144] LustreError: 75277:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010411 with bad export cookie 5570632088982724932 [ 1239.557991] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1239.561077] LustreError: 75277:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1239.690717] Lustre: server umount lustre-MDT0001 complete [ 1241.739350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1244.681759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1249.696700] LustreError: 66157:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000067cfee59 x1839213074000640/t0(0) o250->MGC192.168.203.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1249.807174] LustreError: 74523:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1249.812721] LustreError: 74523:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1249.834400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1249.836408] Lustre: Skipped 1 previous similar message [ 1249.863513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1251.193546] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1254.031225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1254.117533] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.124791] Lustre: Skipped 1 previous similar message [ 1255.490513] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1256.229302] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1257.335974] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1257.952178] Lustre: 66161:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754010413/real 1754010413] req@000000000080067f x1839213074000256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1754010429 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1258.498172] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1261.152157] Lustre: 66158:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754010416/real 1754010416] req@000000008bb87a6c x1839213074000384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1754010432 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1261.539376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1261.540906] Lustre: Skipped 4 previous similar messages [ 1265.632150] Lustre: 66160:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754010421/real 1754010421] req@00000000c93b0bd8 x1839213074000768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1754010437 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1265.644155] Lustre: 66160:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1266.305138] LustreError: 80167:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1266.308235] LustreError: 80167:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1266.338602] Lustre: server umount lustre-MDT0000 complete [ 1266.658530] LustreError: 78779:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.665797] LustreError: 78779:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1267.690101] LustreError: 72909:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010439 with bad export cookie 5570632088982726304 [ 1267.693038] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1267.694232] LustreError: 72909:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1267.848285] Lustre: server umount lustre-MDT0001 complete [ 1270.036602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1271.140765] Lustre: DEBUG MARKER: files compared the same [ 1282.090428] Lustre: server umount lustre-OST0000 complete [ 1288.048660] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1289.072707] Key type lgssc unregistered [ 1289.211311] LNet: 81793:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.278283] LNet: Removed LNI 192.168.203.124@tcp [ 1290.552272] Key type .llcrypt unregistered [ 1290.553292] Key type ._llcrypt unregistered [ 1298.998371] Key type ._llcrypt registered [ 1298.999325] Key type .llcrypt registered [ 1299.034408] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1299.410872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1299.435118] alg: No test for adler32 (adler32-zlib) [ 1300.271430] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 1300.337813] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1300.339539] LNet: Accept secure, port 988 [ 1301.912158] Key type lgssc registered [ 1302.304668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1304.742929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1304.749511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1305.901247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1307.273086] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug +malloc all [ 1310.065507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1310.166692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1311.427326] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug +malloc all [ 1313.119413] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1314.196391] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1316.291850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1316.379469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.174377] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug +malloc all [ 1319.910770] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1330.722755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1339.872534] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1339.874816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1339.877915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1340.896880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1340.897633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1340.899727] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.908071] Lustre: Skipped 1 previous similar message [ 1344.299063] LustreError: 85407:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1344.353920] Lustre: server umount lustre-OST0000 complete [ 1346.016947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1346.016949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.017224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1346.026892] Lustre: Skipped 1 previous similar message [ 1351.136696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1351.140231] Lustre: Skipped 1 previous similar message [ 1351.679838] LustreError: 85609:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1351.682033] LustreError: 85609:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1351.708357] Lustre: server umount lustre-MDT0000 complete [ 1352.995815] LustreError: 85307:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010524 with bad export cookie 13329531387959472115 [ 1352.997456] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1353.000150] LustreError: 85307:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1353.058756] LustreError: 85811:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1353.061056] LustreError: 85811:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1353.133507] Lustre: server umount lustre-MDT0001 complete [ 1357.832743] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1358.880784] Key type lgssc unregistered [ 1359.014357] LNet: 86337:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1360.038638] LNet: Removed LNI 192.168.203.124@tcp [ 1360.325660] Key type .llcrypt unregistered [ 1360.326852] Key type ._llcrypt unregistered [ 1368.910576] Key type ._llcrypt registered [ 1368.912013] Key type .llcrypt registered [ 1368.945013] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1369.354575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1369.360894] alg: No test for adler32 (adler32-zlib) [ 1370.208422] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 1370.291504] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1370.293682] LNet: Accept secure, port 988 [ 1371.872187] Key type lgssc registered [ 1372.205509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1374.766485] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1374.773395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1391.264255] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1397.408150] LustreError: 87672:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.124@tcp: failed processing log, type 4: rc = -110 [ 1427.104159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1428.993078] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1430.740422] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1433.235847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1433.273694] Lustre: *** cfs_fail_loc=706, val=0*** [ 1433.404557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1433.428861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1434.709806] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1437.540881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1437.660635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1439.004193] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1440.905716] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1442.145650] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1443.808968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1443.811524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1443.816579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1446.886585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.887075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1446.894090] Lustre: Skipped 1 previous similar message [ 1452.002073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1452.004317] Lustre: Skipped 1 previous similar message [ 1455.010357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1455.013073] Lustre: Skipped 1 previous similar message [ 1457.120165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1457.138209] LustreError: 89979:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1457.203551] Lustre: server umount lustre-OST0000 complete [ 1460.194681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1460.199735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1460.202200] Lustre: Skipped 1 previous similar message [ 1464.572244] LustreError: 90184:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1464.574377] LustreError: 90184:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1464.602641] Lustre: server umount lustre-MDT0000 complete [ 1465.313220] LustreError: 88642:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1465.317879] LustreError: 88642:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1465.838178] LustreError: 87681:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010637 with bad export cookie 4588396892659445469 [ 1465.840995] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1465.841277] LustreError: 87681:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1465.904170] LustreError: 90386:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1465.906507] LustreError: 90386:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1465.986621] Lustre: server umount lustre-MDT0001 complete [ 1469.645603] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1470.724526] Key type lgssc unregistered [ 1470.863351] LNet: 90912:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1471.910389] LNet: Removed LNI 192.168.203.124@tcp [ 1472.235410] Key type .llcrypt unregistered [ 1472.236787] Key type ._llcrypt unregistered [ 1481.287159] Key type ._llcrypt registered [ 1481.288470] Key type .llcrypt registered [ 1481.326785] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1481.725592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1481.755465] alg: No test for adler32 (adler32-zlib) [ 1482.612639] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 1482.689863] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1482.691731] LNet: Accept secure, port 988 [ 1484.280166] Key type lgssc registered [ 1484.639858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1487.278763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1487.286455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1489.531557] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1491.895328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1492.049064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1493.230540] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1495.929438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1496.055614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1497.875674] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1500.537913] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1500.541171] Lustre: MGS: Client fd479e8e-20da-4744-a64e-f7f72b9ec64a (at 0@lo) reconnecting [ 1500.619187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1500.655957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1501.911817] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1510.881395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1510.883617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1510.887831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1511.905872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.906809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1511.907158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1511.914114] Lustre: Skipped 1 previous similar message [ 1515.692068] LustreError: 94880:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1515.753067] Lustre: server umount lustre-OST0000 complete [ 1517.025588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1517.025596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1517.026262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1517.033328] Lustre: Skipped 1 previous similar message [ 1522.146085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1522.147701] Lustre: Skipped 2 previous similar messages [ 1527.265483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1527.267392] Lustre: Skipped 1 previous similar message [ 1531.360153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1531.394609] LustreError: 95082:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1531.396670] LustreError: 95082:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1531.409594] Lustre: server umount lustre-MDT0000 complete [ 1532.773820] LustreError: 95285:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1532.776235] LustreError: 95285:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1532.848745] Lustre: server umount lustre-MDT0001 complete [ 1534.051838] Lustre: server umount MGS complete [ 1537.756356] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 21:11:49 (1754010709) [ 1546.769431] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 1549.657779] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1552.654653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1554.758383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1556.614683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1558.615863] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1560.706919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1560.727498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1561.933404] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1565.757632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1565.784046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1565.862736] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1565.879697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1565.915582] Lustre: lustre-MDT0001: new disk, initializing [ 1565.942111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1565.951519] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1565.954527] Lustre: 99972:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1565.958694] LustreError: 99972:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1567.208495] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1571.272522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1571.300588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1571.382944] Lustre: lustre-OST0000: new disk, initializing [ 1571.384922] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1571.404059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1573.373142] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1576.618634] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1578.978123] Lustre: 101474:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@00000000658ebf61 x1839213475563136/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1754010766 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1579.002644] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1579.006979] Lustre: MGS: Client 4e93cdf0-e1a2-44ba-94a6-53310736e32f (at 0@lo) reconnecting [ 1579.020697] Lustre: 99264:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1579.027787] LustreError: 99264:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1579.037665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1579.076919] Lustre: lustre-MDT0000: new disk, initializing [ 1579.140166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1580.193067] Lustre: 101661:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@000000005df16525 x1839213475568256/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1754010768 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1580.199806] Lustre: 101661:0:(client.c:1611:after_reply()) Skipped 1 previous similar message [ 1580.381137] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1581.024590] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1588.067098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1588.070276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1589.282863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1589.285810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1589.286734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1609.696923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1609.700280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.704137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1609.706712] Lustre: Skipped 2 previous similar messages [ 1615.214423] LustreError: 102190:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1615.216774] LustreError: 102190:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1615.279134] Lustre: server umount lustre-OST0000 complete [ 1616.865849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1616.865917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1616.871823] Lustre: Skipped 2 previous similar messages [ 1622.783160] LustreError: 102394:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1622.786247] LustreError: 102394:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1622.797934] Lustre: server umount lustre-MDT0000 complete [ 1638.368132] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1638.396509] LustreError: 102595:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1638.398054] LustreError: 102595:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1638.467917] Lustre: server umount lustre-MDT0001 complete [ 1639.655258] Lustre: server umount MGS complete [ 1643.268707] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 21:13:34 (1754010814) [ 1650.193358] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1651.353297] Key type lgssc unregistered [ 1651.500341] LNet: 104126:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.518573] LNet: Removed LNI 192.168.203.124@tcp [ 1652.865081] Key type .llcrypt unregistered [ 1652.866957] Key type ._llcrypt unregistered [ 1660.045770] Key type ._llcrypt registered [ 1660.047582] Key type .llcrypt registered [ 1660.087876] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1660.374721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1660.387663] alg: No test for adler32 (adler32-zlib) [ 1661.262506] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 1661.354895] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1661.356542] LNet: Accept secure, port 988 [ 1662.944166] Key type lgssc registered [ 1663.329582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1666.092696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1666.100455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1667.257641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1668.530542] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1671.343372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1671.461619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1672.702401] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1674.427730] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1675.586821] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1677.789225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1677.895123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1679.643478] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1681.292526] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1692.262793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1702.881908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.883428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1702.885497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1702.888900] Lustre: Skipped 1 previous similar message [ 1705.773222] LustreError: 107545:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1705.836965] Lustre: server umount lustre-OST0000 complete [ 1707.488889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1707.491349] LustreError: Skipped 1 previous similar message [ 1707.492652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1707.496859] Lustre: Skipped 1 previous similar message [ 1707.498698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1713.121494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1713.124332] Lustre: Skipped 2 previous similar messages [ 1713.283711] LustreError: 107747:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1713.288049] LustreError: 107747:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1713.319916] Lustre: server umount lustre-MDT0000 complete [ 1714.701640] LustreError: 106009:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010886 with bad export cookie 108751803448062227 [ 1714.704232] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1714.706499] LustreError: 106009:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1714.780022] LustreError: 107948:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1714.783493] LustreError: 107948:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1714.870621] Lustre: server umount lustre-MDT0001 complete [ 1719.279168] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1720.238470] Key type lgssc unregistered [ 1720.372342] LNet: 108473:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1721.382256] LNet: Removed LNI 192.168.203.124@tcp [ 1721.671676] Key type .llcrypt unregistered [ 1721.672705] Key type ._llcrypt unregistered [ 1727.590386] Key type ._llcrypt registered [ 1727.591572] Key type .llcrypt registered [ 1727.630269] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1728.002359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1728.031667] alg: No test for adler32 (adler32-zlib) [ 1728.893115] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 1728.982858] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1728.985652] LNet: Accept secure, port 988 [ 1730.568174] Key type lgssc registered [ 1730.948706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1733.079836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1733.080709] LustreError: 109503:0:(tgt_mount.c:2375:server_fill_super()) cfs_race id 716 sleeping [ 1733.083697] Lustre: Skipped 1 previous similar message [ 1733.087185] LustreError: 109504:0:(tgt_mount.c:2375:server_fill_super()) cfs_fail_race id 716 waking [ 1733.089112] LustreError: 109503:0:(tgt_mount.c:2375:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1733.091461] LustreError: 109503:0:(tgt_mount.c:2387:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1733.095510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1733.095888] LustreError: 109503:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1734.253701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1736.782698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1736.876986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1738.156461] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1739.786051] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing lsmod [ 1740.793086] LustreError: 110667:0:(tgt_mount.c:2375:server_fill_super()) cfs_race id 716 sleeping [ 1740.793456] LustreError: 110668:0:(tgt_mount.c:2375:server_fill_super()) cfs_fail_race id 716 waking [ 1740.799389] LustreError: 110667:0:(tgt_mount.c:2375:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1740.802121] LustreError: 110667:0:(tgt_mount.c:2387:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1740.805677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1740.806604] LustreError: 110667:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1740.928895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1741.914623] LustreError: 109573:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1741.918577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.918993] LustreError: 109573:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1741.919690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1741.922851] Lustre: Skipped 2 previous similar messages [ 1741.926568] LustreError: 109573:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1747.042252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1747.045575] Lustre: Skipped 4 previous similar messages [ 1748.028717] LustreError: 110856:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1748.060739] Lustre: server umount lustre-MDT0000 complete [ 1749.305145] LustreError: 109519:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010921 with bad export cookie 10996449105308272933 [ 1749.308753] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1749.308876] LustreError: 109519:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1751.136232] LustreError: 109977:0:(client.c:1375:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008608dfe1 x1839213733493888/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1751.145123] LustreError: 111060:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1751.148050] LustreError: 111060:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1751.218631] Lustre: server umount lustre-MDT0001 complete [ 1762.801334] LustreError: 111261:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1762.803507] LustreError: 111261:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1762.871582] Lustre: server umount lustre-OST0000 complete [ 1765.865753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.028640] LustreError: 111738:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1766.039411] LustreError: 111738:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1766.056745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1767.433759] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1770.535053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1770.668495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1772.076701] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1773.794501] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1774.916708] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1776.981748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1778.874182] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1780.542353] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1790.440262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1795.553750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1795.555867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1795.557413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1795.562622] Lustre: Skipped 1 previous similar message [ 1798.767090] LustreError: 113987:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1798.769980] LustreError: 113987:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1798.838082] Lustre: server umount lustre-OST0000 complete [ 1800.673530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.674161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1800.680829] Lustre: Skipped 1 previous similar message [ 1800.683151] Lustre: Skipped 1 previous similar message [ 1805.793208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1805.795653] Lustre: Skipped 1 previous similar message [ 1806.269583] LustreError: 114188:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1806.271826] LustreError: 114188:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1806.304478] Lustre: server umount lustre-MDT0000 complete [ 1807.602144] LustreError: 111718:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010979 with bad export cookie 10996449105308273612 [ 1807.604988] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1807.605939] LustreError: 111718:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1807.750567] Lustre: server umount lustre-MDT0001 complete [ 1812.585213] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1813.625765] Key type lgssc unregistered [ 1813.766297] LNet: 114916:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.822281] LNet: Removed LNI 192.168.203.124@tcp [ 1815.108584] Key type .llcrypt unregistered [ 1815.110037] Key type ._llcrypt unregistered [ 1823.397958] Key type ._llcrypt registered [ 1823.399363] Key type .llcrypt registered [ 1823.435681] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1823.761576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1823.824259] alg: No test for adler32 (adler32-zlib) [ 1824.676997] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 1824.753322] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1824.755190] LNet: Accept secure, port 988 [ 1826.336129] Key type lgssc registered [ 1826.672418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1829.135628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1829.141582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.288917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1831.524129] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1834.190430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1834.305073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1835.642643] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1837.369907] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1838.513493] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1840.664172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1840.761090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1842.490698] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1844.131883] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1856.103319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1861.950761] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1862.778412] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1865.696830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1865.699359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.703549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1866.723278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1866.723959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1866.725664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1869.871044] LustreError: 118630:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1869.932573] Lustre: server umount lustre-OST0000 complete [ 1871.841364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1871.841800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1871.845132] Lustre: Skipped 1 previous similar message [ 1871.850248] Lustre: Skipped 2 previous similar messages [ 1876.961086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1876.963136] Lustre: Skipped 1 previous similar message [ 1877.370271] LustreError: 118832:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1877.371813] LustreError: 118832:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1877.396602] Lustre: server umount lustre-MDT0000 complete [ 1878.561126] LustreError: 118482:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011050 with bad export cookie 5265968170242652234 [ 1878.563790] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1878.565692] LustreError: 118482:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1878.632124] LustreError: 119037:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1878.633897] LustreError: 119037:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1878.704895] Lustre: server umount lustre-MDT0001 complete [ 1883.144937] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1884.240815] Key type lgssc unregistered [ 1884.378437] LNet: 119562:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.414303] LNet: Removed LNI 192.168.203.124@tcp [ 1885.729889] Key type .llcrypt unregistered [ 1885.731593] Key type ._llcrypt unregistered [ 1892.810838] Key type ._llcrypt registered [ 1892.812009] Key type .llcrypt registered [ 1892.856190] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1893.222439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1893.265462] alg: No test for adler32 (adler32-zlib) [ 1894.114536] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 1894.191212] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1894.192658] LNet: Accept secure, port 988 [ 1895.768148] Key type lgssc registered [ 1896.145874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1898.837471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1898.843582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.007943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.340565] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1904.243110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1904.377617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1905.733823] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1907.561614] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1908.660911] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1910.821974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1910.933282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.645027] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1914.292273] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1924.267498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1929.697719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1929.698725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.700455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1929.708076] Lustre: Skipped 1 previous similar message [ 1933.678612] LustreError: 122982:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1933.738777] Lustre: server umount lustre-OST0000 complete [ 1935.328767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1935.330904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1935.334799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1935.336529] Lustre: Skipped 1 previous similar message [ 1939.937104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1939.937509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1941.049205] LustreError: 123185:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1941.051776] LustreError: 123185:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1941.071572] Lustre: server umount lustre-MDT0000 complete [ 1942.238103] LustreError: 120710:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011114 with bad export cookie 881378750239496259 [ 1942.241759] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1942.245207] LustreError: 120710:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1942.303372] LustreError: 123387:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1942.305077] LustreError: 123387:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1942.378232] Lustre: server umount lustre-MDT0001 complete [ 1946.850427] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1947.924807] Key type lgssc unregistered [ 1948.051378] LNet: 123913:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1949.094298] LNet: Removed LNI 192.168.203.124@tcp [ 1949.386155] Key type .llcrypt unregistered [ 1949.387665] Key type ._llcrypt unregistered [ 1957.788415] Key type ._llcrypt registered [ 1957.789783] Key type .llcrypt registered [ 1957.825841] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1958.171385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1958.206060] alg: No test for adler32 (adler32-zlib) [ 1959.060674] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 1959.144952] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1959.146339] LNet: Accept secure, port 988 [ 1960.728173] Key type lgssc registered [ 1961.067454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1963.532102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1963.538829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.680628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1965.841544] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1968.490278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1968.609500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1969.800620] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1971.583724] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1972.671944] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1974.852986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1974.958550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1976.720537] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1978.483639] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1989.286211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1996.602770] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1997.843878] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1997.846348] Lustre: Skipped 2 previous similar messages [ 1999.394613] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1999.397166] Lustre: Skipped 2 previous similar messages [ 2003.452844] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2003.463558] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2008.736232] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.24@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2008.739671] Lustre: Skipped 2 previous similar messages [ 2013.687452] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.24@tcp 192.168.0.[2,10]@tcp [ 2015.388700] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.24/24@tcp in log lustre-MDT0000 [ 2015.392629] Lustre: Skipped 2 previous similar messages [ 2022.904253] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.24/24@tcp [ 2022.906582] Lustre: Skipped 1 previous similar message [ 2025.441726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2025.442027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.445095] LustreError: Skipped 1 previous similar message [ 2025.448089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2025.450786] Lustre: Skipped 1 previous similar message [ 2030.561354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2030.563952] Lustre: Skipped 1 previous similar message [ 2030.764060] LustreError: 129167:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2030.819707] Lustre: server umount lustre-OST0000 complete [ 2035.169109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2035.171414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2035.174290] Lustre: Skipped 1 previous similar message [ 2035.175682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2038.142389] LustreError: 129368:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2038.144843] LustreError: 129368:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2038.172261] Lustre: server umount lustre-MDT0000 complete [ 2039.445267] LustreError: 125257:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011211 with bad export cookie 5831200833495296685 [ 2039.447711] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2039.449958] LustreError: 125257:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2039.515610] LustreError: 129570:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2039.517438] LustreError: 129570:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2039.598134] Lustre: server umount lustre-MDT0001 complete [ 2044.331158] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2045.401150] Key type lgssc unregistered [ 2045.534408] LNet: 130096:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.566309] LNet: Removed LNI 192.168.203.124@tcp [ 2046.862760] Key type .llcrypt unregistered [ 2046.863694] Key type ._llcrypt unregistered [ 2057.105939] Key type ._llcrypt registered [ 2057.107245] Key type .llcrypt registered [ 2057.144335] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2057.536138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2057.559751] alg: No test for adler32 (adler32-zlib) [ 2058.412432] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 2058.493792] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2058.496233] LNet: Accept secure, port 988 [ 2060.072130] Key type lgssc registered [ 2060.398712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2062.839625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2062.848079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.000718] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2064.005545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.010897] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.24/24@tcp [ 2065.246728] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2067.975078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2068.088309] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2068.098261] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.24/24@tcp [ 2068.110751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2069.482615] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2071.244567] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2072.496664] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2074.860067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2074.987788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.869851] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2078.604673] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2088.296325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2093.537648] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2093.538975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2093.540157] LustreError: Skipped 1 previous similar message [ 2093.541074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2093.543893] Lustre: Skipped 1 previous similar message [ 2094.445891] LustreError: 133926:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2094.521593] Lustre: server umount lustre-OST0000 complete [ 2098.658973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.659692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.662597] Lustre: Skipped 1 previous similar message [ 2098.664431] Lustre: Skipped 1 previous similar message [ 2101.949488] LustreError: 134127:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2101.951299] LustreError: 134127:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2101.970314] Lustre: server umount lustre-MDT0000 complete [ 2103.285221] LustreError: 131604:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011275 with bad export cookie 3863590380083231018 [ 2103.288582] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2103.289807] LustreError: 131604:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2103.352854] LustreError: 134328:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2103.354864] LustreError: 134328:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2103.435694] Lustre: server umount lustre-MDT0001 complete [ 2108.255262] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2109.483247] Key type lgssc unregistered [ 2109.626401] LNet: 134854:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2110.694457] LNet: Removed LNI 192.168.203.124@tcp [ 2111.043970] Key type .llcrypt unregistered [ 2111.045201] Key type ._llcrypt unregistered [ 2119.809761] Key type ._llcrypt registered [ 2119.811436] Key type .llcrypt registered [ 2119.855329] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 2125.472637] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2125.778926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2125.787955] alg: No test for adler32 (adler32-zlib) [ 2126.630301] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 2126.707182] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2126.708615] LNet: Accept secure, port 988 [ 2128.288154] Key type lgssc registered [ 2128.626691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2131.370821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2133.424894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2135.243432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2137.088947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2139.142414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2139.159085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2139.163458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2140.238583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2140.248448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2140.280201] Lustre: lustre-MDT0000: new disk, initializing [ 2140.300717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2140.306605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2141.523907] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2145.772556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2145.797741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2145.830835] Lustre: 138184:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2145.856783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2145.859527] Lustre: Skipped 1 previous similar message [ 2145.903574] Lustre: lustre-MDT0001: new disk, initializing [ 2145.940059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2145.952728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2145.956261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2147.303581] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2150.440308] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2151.630480] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2153.846901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2153.872307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2153.973830] Lustre: lustre-OST0000: new disk, initializing [ 2153.977111] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2154.013608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2155.448427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2155.452148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2155.491321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2155.968481] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2159.177977] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2160.532442] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2160.601865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2162.027824] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2162.102941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2165.729701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2165.731384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2165.733604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2165.737929] Lustre: Skipped 1 previous similar message [ 2165.740945] Lustre: Skipped 1 previous similar message [ 2168.877945] LustreError: 141091:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2168.938826] Lustre: server umount lustre-OST0000 complete [ 2170.849389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2170.849961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2170.854068] Lustre: Skipped 1 previous similar message [ 2175.969495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2175.972112] Lustre: Skipped 2 previous similar messages [ 2176.317168] LustreError: 141292:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2176.319042] LustreError: 141292:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2176.344554] Lustre: server umount lustre-MDT0000 complete [ 2177.596184] LustreError: 138174:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011349 with bad export cookie 5986591063572864292 [ 2177.599499] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2177.600656] LustreError: 138174:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2177.674122] LustreError: 141493:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2177.675852] LustreError: 141493:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2177.758828] Lustre: server umount lustre-MDT0001 complete [ 2180.650355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2180.847236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2182.225561] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2185.163693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2185.283171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2186.670842] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2188.572256] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2189.835815] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2192.179263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2194.319362] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2196.288823] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2197.767032] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2207.052412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2208.395666] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2208.468914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2213.203116] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2213.244454] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2213.354385] Lustre: lustre-OST0001: new disk, initializing [ 2213.357840] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2213.443341] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2213.448063] Lustre: Skipped 1 previous similar message [ 2217.236225] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2219.651724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2219.681360] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2219.775457] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2223.864977] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2224.050255] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2227.559486] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2227.734974] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2231.096688] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d3685d21000.ost_server_uuid 50 [ 2232.534254] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d3685d21000.ost_server_uuid in FULL state after 0 sec [ 2240.481064] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2240.484719] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.498644] LustreError: Skipped 2 previous similar messages [ 2240.502620] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2240.507702] Lustre: Skipped 1 previous similar message [ 2240.540233] Lustre: Skipped 1 previous similar message [ 2245.435230] LustreError: 146226:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2245.438993] LustreError: 146226:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2245.604551] LustreError: 143552:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2245.619662] Lustre: server umount lustre-OST0001 complete [ 2245.634560] LustreError: 143552:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2250.723835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2250.728618] LustreError: 143554:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.728918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2250.739504] Lustre: Skipped 1 previous similar message [ 2250.782361] LustreError: 143554:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2255.095502] LustreError: 146427:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2255.115706] LustreError: 146427:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2255.251563] Lustre: server umount lustre-OST0000 complete [ 2260.962725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2260.970754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2260.973399] Lustre: Skipped 1 previous similar message [ 2272.224214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2272.267235] LustreError: 146629:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2272.269352] LustreError: 146629:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2272.301326] Lustre: server umount lustre-MDT0000 complete [ 2273.642458] LustreError: 141950:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011445 with bad export cookie 5986591063572865160 [ 2273.645352] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2273.646805] LustreError: 141950:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2273.830378] Lustre: server umount lustre-MDT0001 complete [ 2282.176925] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2283.282634] Key type lgssc unregistered [ 2283.420363] LNet: 147815:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2284.455208] LNet: Removed LNI 192.168.203.124@tcp [ 2284.777345] Key type .llcrypt unregistered [ 2284.779063] Key type ._llcrypt unregistered [ 2298.070876] Key type ._llcrypt registered [ 2298.072455] Key type .llcrypt registered [ 2298.119371] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 2305.379490] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2305.820847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2305.872512] alg: No test for adler32 (adler32-zlib) [ 2306.747471] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 2306.851866] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2306.853862] LNet: Accept secure, port 988 [ 2308.440156] Key type lgssc registered [ 2308.883181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2312.553292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2315.243046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2317.590795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2319.983247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2322.556585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2322.575336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2322.581296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.672958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2323.686236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2323.727628] Lustre: lustre-MDT0000: new disk, initializing [ 2323.758290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2323.767855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2325.244678] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2329.636685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2329.661731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2329.699292] Lustre: 151720:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2329.724230] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2329.726573] Lustre: Skipped 1 previous similar message [ 2329.763517] Lustre: lustre-MDT0001: new disk, initializing [ 2329.798469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2329.812090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2329.815775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2331.193206] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2334.309336] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2335.662951] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2338.379031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2338.420584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2338.555380] Lustre: lustre-OST0000: new disk, initializing [ 2338.559178] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2338.598506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2340.999750] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2343.422504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2343.427089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2343.456909] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2344.474095] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2346.096271] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2346.191743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2347.643679] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2347.727157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2348.514947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2348.515567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2348.516186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2348.518351] LustreError: Skipped 1 previous similar message [ 2348.523122] Lustre: Skipped 1 previous similar message [ 2350.458882] LustreError: 154625:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2350.563322] Lustre: server umount lustre-OST0000 complete [ 2353.634379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2353.637261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2353.642022] Lustre: Skipped 1 previous similar message [ 2353.648604] Lustre: Skipped 2 previous similar messages [ 2358.452393] LustreError: 154826:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2358.465264] LustreError: 154826:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2358.656485] Lustre: server umount lustre-MDT0000 complete [ 2362.095673] LustreError: 151711:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011533 with bad export cookie 1282223733436175732 [ 2362.100966] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2362.109978] LustreError: 151711:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2362.213455] LustreError: 155027:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2362.216336] LustreError: 155027:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2362.384437] Lustre: server umount lustre-MDT0001 complete [ 2367.864268] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 21:25:38 (1754011538) [ 2377.773760] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 2381.903714] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2386.090835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2389.054146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2391.633811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2394.170428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2397.140434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2397.189679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2397.291746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2397.309974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2397.355494] Lustre: lustre-MDT0000: new disk, initializing [ 2397.411342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.420795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2399.118278] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2404.235041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2404.264644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2404.307423] Lustre: 158000:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2404.335154] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2404.337693] Lustre: Skipped 1 previous similar message [ 2404.376309] Lustre: lustre-MDT0001: new disk, initializing [ 2404.417437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2404.433400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2404.437451] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2406.120127] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2409.893788] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2411.505439] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2414.462342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2414.501038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2414.619187] Lustre: lustre-OST0000: new disk, initializing [ 2414.621700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2414.655912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2417.197413] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2420.902429] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2424.846318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2424.851220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2424.892888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2428.314901] Lustre: Failing over lustre-OST0000 [ 2428.351853] LustreError: 160421:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2428.354968] LustreError: 160421:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2428.455722] Lustre: server umount lustre-OST0000 complete [ 2442.229104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2442.376787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2442.381754] Lustre: Skipped 1 previous similar message [ 2442.392124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2442.401109] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2442.614842] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2444.841294] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2445.473124] Lustre: 149940:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754011601/real 1754011601] req@0000000061f99f99 x1839214339611392/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1754011617 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2446.304207] Lustre: 149942:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754011601/real 1754011601] req@0000000029b2b008 x1839214339611648/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1754011617 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2446.499884] Lustre: Failing over lustre-MDT0000 [ 2446.566553] LustreError: 161386:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2446.568753] LustreError: 161386:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2446.609678] Lustre: server umount lustre-MDT0000 complete [ 2447.733640] LustreError: 160325:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.743664] LustreError: 160325:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2447.841979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2447.848642] Lustre: Skipped 1 previous similar message [ 2450.912154] Lustre: 149943:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754011606/real 1754011606] req@00000000368bf487 x1839214339613312/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1754011622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2452.838069] LustreError: 158005:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.845508] LustreError: 158005:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2456.032379] Lustre: 149943:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754011611/real 1754011611] req@00000000ba56de07 x1839214339614336/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1754011627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2457.965208] LustreError: 158005:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.974963] LustreError: 158005:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2460.676871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2460.746786] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2460.756403] Lustre: 158003:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.203.124@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2460.869563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2460.914994] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2462.621759] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2463.082700] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2466.279177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2466.287694] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2466.315674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2471.396632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2471.397462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.398560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2471.400452] LustreError: Skipped 1 previous similar message [ 2471.413013] Lustre: Skipped 1 previous similar message [ 2473.645068] LustreError: 162262:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2473.650024] LustreError: 162262:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2473.724332] Lustre: server umount lustre-OST0000 complete [ 2476.514994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2476.515769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2476.524402] Lustre: Skipped 2 previous similar messages [ 2481.342543] Lustre: server umount lustre-MDT0000 complete [ 2481.634143] LustreError: 158011:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.641602] LustreError: 158011:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 2483.024412] LustreError: 158775:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011654 with bad export cookie 1282223733436177790 [ 2483.025558] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2483.030485] LustreError: 158775:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2483.210926] Lustre: server umount lustre-MDT0001 complete [ 2488.560606] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2489.883882] Key type lgssc unregistered [ 2490.037356] LNet: 163192:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2491.111031] LNet: Removed LNI 192.168.203.124@tcp [ 2491.470184] Key type .llcrypt unregistered [ 2491.471975] Key type ._llcrypt unregistered [ 2513.027947] Key type ._llcrypt registered [ 2513.029014] Key type .llcrypt registered [ 2513.071858] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2513.463444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2513.491088] alg: No test for adler32 (adler32-zlib) [ 2514.378817] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 2514.483592] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2514.485685] LNet: Accept secure, port 988 [ 2516.080291] Key type lgssc registered [ 2516.499240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2519.639240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2519.649354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2520.877243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2522.399994] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2525.990429] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2528.136187] LustreError: 164708:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2528.140626] LustreError: 164708:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2528.143326] LustreError: 164708:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2528.214847] LustreError: 165225:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2528.320805] Lustre: server umount lustre-MDT0000 complete [ 2534.084817] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 2537.781472] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2541.779767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2544.293533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2546.496538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2548.933000] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2551.395979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2551.420749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2551.510627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2551.527511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2551.566825] Lustre: lustre-MDT0000: new disk, initializing [ 2551.612568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2551.621543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2553.157904] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2557.643228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2557.668135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2557.704098] Lustre: 167848:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2557.733293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2557.735376] Lustre: Skipped 1 previous similar message [ 2557.777095] Lustre: lustre-MDT0001: new disk, initializing [ 2557.816550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2557.833905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2557.837785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2559.231119] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2562.534141] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2563.903043] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2566.599502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2566.638663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2566.762900] Lustre: lustre-OST0000: new disk, initializing [ 2566.765938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2566.808943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2569.339196] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2572.804820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2572.808868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2572.869702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2573.032833] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2576.161854] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2576.912808] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 21:29:08 (1754011748) [ 2680.288891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2680.289122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2680.294994] Lustre: Skipped 2 previous similar messages [ 2680.297654] Lustre: Skipped 1 previous similar message [ 2684.796490] LustreError: 171338:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2684.798919] LustreError: 171338:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2684.862858] Lustre: server umount lustre-MDT0000 complete [ 2685.409453] LustreError: 167858:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2685.416740] LustreError: 167858:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 2686.340119] LustreError: 167840:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011858 with bad export cookie 2812610944131910171 [ 2686.340686] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2686.344768] LustreError: 167840:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2690.528746] LustreError: 171232:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2690.528908] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.540083] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2690.542817] Lustre: Skipped 1 previous similar message [ 2691.552628] LustreError: 171232:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.552885] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2692.480901] LustreError: 171539:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2692.482921] LustreError: 171539:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2692.580944] Lustre: server umount lustre-MDT0001 complete [ 2700.268941] LustreError: 171740:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2700.272132] LustreError: 171740:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2700.333630] Lustre: server umount lustre-OST0000 complete [ 2703.780595] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 2707.303604] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2711.328796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2714.151961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2716.590049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2719.131905] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2721.938081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2721.965204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.059291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2722.074591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2722.116289] Lustre: lustre-MDT0000: new disk, initializing [ 2722.147429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2722.156704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2723.613671] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2728.192662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2728.221460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2728.259637] Lustre: 174059:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2728.287427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2728.289840] Lustre: Skipped 1 previous similar message [ 2728.332445] Lustre: lustre-MDT0001: new disk, initializing [ 2728.388605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2728.392992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2729.945731] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2733.577475] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2734.958338] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2737.441036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2737.469089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2737.570956] Lustre: lustre-OST0000: new disk, initializing [ 2737.572887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2737.603360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2737.606195] Lustre: Skipped 1 previous similar message [ 2739.636259] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2740.100813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2740.105039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2740.146665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2742.981721] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2744.518397] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2744.614155] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2746.207041] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2746.295550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2750.434305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2750.436304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2750.438351] LustreError: Skipped 1 previous similar message [ 2750.444232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2750.449467] Lustre: Skipped 1 previous similar message [ 2753.071115] LustreError: 176967:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2753.074119] LustreError: 176967:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2753.144177] Lustre: server umount lustre-OST0000 complete [ 2755.554127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.555376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2755.559164] Lustre: Skipped 2 previous similar messages [ 2755.562967] Lustre: Skipped 1 previous similar message [ 2760.735412] Lustre: server umount lustre-MDT0000 complete [ 2762.254396] LustreError: 174051:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011934 with bad export cookie 2812610944132068987 [ 2762.257680] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2762.260357] LustreError: 174051:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2762.338873] LustreError: 177370:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2762.342251] LustreError: 177370:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2762.471724] Lustre: server umount lustre-MDT0001 complete [ 2765.889415] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:32:17 (1754011937) [ 2771.832882] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 2775.139529] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2778.758984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2781.241750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2783.546073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2786.011178] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2788.862184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2788.898826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2789.007046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2789.027270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2789.068091] Lustre: lustre-MDT0000: new disk, initializing [ 2789.100308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2789.109666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2790.648213] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2795.491030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2795.521993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2795.574567] Lustre: 180337:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2795.698851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2797.277741] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2800.834216] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2802.481080] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2805.178995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2805.210294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2807.042553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2807.088742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2807.700727] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2811.140150] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2817.508236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2817.508328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2817.511024] LustreError: Skipped 1 previous similar message [ 2817.516553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2817.519631] Lustre: Skipped 3 previous similar messages [ 2819.437883] LustreError: 182865:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2819.440342] LustreError: 182865:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2819.508622] Lustre: server umount lustre-OST0000 complete [ 2821.601632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2827.165513] Lustre: server umount lustre-MDT0000 complete [ 2827.745048] LustreError: 181123:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.751782] LustreError: 181123:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2828.726641] LustreError: 180330:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012000 with bad export cookie 2812610944132069855 [ 2828.729521] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2828.731111] LustreError: 180330:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2828.895821] Lustre: server umount lustre-MDT0001 complete [ 2832.399690] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:33:23 (1754012003) [ 2838.444576] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 2841.819667] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2845.506785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2848.180101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2850.578418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2853.118777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2855.883743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2855.913135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2856.026774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2856.043116] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2856.045902] Lustre: Skipped 3 previous similar messages [ 2856.088124] Lustre: lustre-MDT0000: new disk, initializing [ 2856.090088] Lustre: Skipped 2 previous similar messages [ 2856.126236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2856.129073] Lustre: Skipped 2 previous similar messages [ 2856.137205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2856.141296] Lustre: Skipped 2 previous similar messages [ 2857.767714] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2862.524067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2862.560233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2862.607133] Lustre: 186239:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2862.643184] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2862.645827] Lustre: Skipped 3 previous similar messages [ 2862.743145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2864.266682] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2867.621276] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2869.032259] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2871.659618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2871.685607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2873.947996] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2875.040208] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2877.349103] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2880.484134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2880.485140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2880.487454] LustreError: Skipped 1 previous similar message [ 2880.491972] Lustre: Skipped 3 previous similar messages [ 2885.548118] LustreError: 188777:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2885.550526] LustreError: 188777:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2885.619254] Lustre: server umount lustre-OST0000 complete [ 2888.673295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2893.143707] Lustre: server umount lustre-MDT0000 complete [ 2894.599922] LustreError: 187993:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012066 with bad export cookie 2812610944132070940 [ 2894.602798] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2894.605101] LustreError: 187993:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2894.760572] Lustre: server umount lustre-MDT0001 complete [ 2899.927500] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2901.151432] Key type lgssc unregistered [ 2901.313715] LNet: 189707:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2902.374420] LNet: Removed LNI 192.168.203.124@tcp [ 2902.739216] Key type .llcrypt unregistered [ 2902.740799] Key type ._llcrypt unregistered [ 2912.435813] Key type ._llcrypt registered [ 2912.437430] Key type .llcrypt registered [ 2912.481676] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2912.771549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2912.809448] alg: No test for adler32 (adler32-zlib) [ 2913.686359] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 2913.789646] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2913.792425] LNet: Accept secure, port 988 [ 2915.376168] Key type lgssc registered [ 2915.773178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2918.840791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2918.849594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2920.038431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2921.538626] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2924.755692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2924.899222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2926.297199] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2928.239288] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2929.571872] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2932.105268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2932.242820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.346858] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2936.430184] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2944.548817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2959.843249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2959.843826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2959.845789] LustreError: Skipped 1 previous similar message [ 2959.852557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2959.856641] Lustre: Skipped 1 previous similar message [ 2964.717171] LustreError: 193333:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2964.776806] Lustre: server umount lustre-OST0000 complete [ 2970.081373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2970.081843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2970.085954] Lustre: Skipped 2 previous similar messages [ 2972.221512] LustreError: 193534:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2972.224279] LustreError: 193534:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2972.251496] Lustre: server umount lustre-MDT0000 complete [ 2973.558144] LustreError: 191063:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012145 with bad export cookie 5540749156422398888 [ 2973.561371] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2973.562035] LustreError: 191063:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2973.625259] LustreError: 193737:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2973.628063] LustreError: 193737:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2973.708039] Lustre: server umount lustre-MDT0001 complete [ 2978.463950] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2979.577322] Key type lgssc unregistered [ 2979.720453] LNet: 194264:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2980.774690] LNet: Removed LNI 192.168.203.124@tcp [ 2981.140838] Key type .llcrypt unregistered [ 2981.142378] Key type ._llcrypt unregistered [ 2991.096207] Key type ._llcrypt registered [ 2991.097548] Key type .llcrypt registered [ 2991.145468] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2991.472033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2991.486207] alg: No test for adler32 (adler32-zlib) [ 2992.343765] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 2992.435821] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2992.438106] LNet: Accept secure, port 988 [ 2994.016150] Key type lgssc registered [ 2994.375970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2997.217981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2997.226484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.410394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2999.756902] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3002.737188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3002.859108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3004.187254] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3006.123386] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3007.499067] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3010.071539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3010.210311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3012.466579] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3014.460882] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3023.529392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3027.937158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3027.940194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3027.945469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3028.961570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.961875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3028.963116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3028.972164] Lustre: Skipped 1 previous similar message [ 3033.133118] LustreError: 197872:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3033.196480] Lustre: server umount lustre-OST0000 complete [ 3035.221472] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8d3698142800.ost_server_uuid 50 [ 3035.868417] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3045.903230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.903675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3045.907961] Lustre: Skipped 1 previous similar message [ 3045.910744] Lustre: Skipped 1 previous similar message [ 3050.977768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3050.980411] Lustre: Skipped 1 previous similar message [ 3052.030537] LustreError: 198220:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3052.032415] LustreError: 198220:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3052.061098] Lustre: server umount lustre-MDT0000 complete [ 3053.375482] LustreError: 198222:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012225 with bad export cookie 15808265985369994809 [ 3053.379194] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3053.379528] LustreError: 198222:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3053.451518] LustreError: 198423:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3053.454184] LustreError: 198423:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3053.547185] Lustre: server umount lustre-MDT0001 complete [ 3056.687693] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 21:37:08 (1754012228) [ 3059.193962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.369327] LustreError: 199093:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3059.377143] LustreError: 199093:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3059.394587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3060.863072] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3064.042608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3064.202356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3065.613810] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3067.634825] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3068.895509] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3071.217417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3073.303391] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3075.131594] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3077.517138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3077.549455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3077.615963] Lustre: lustre-OST0001: new disk, initializing [ 3077.619078] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3077.652777] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3077.654905] Lustre: Skipped 1 previous similar message [ 3079.679112] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3082.884271] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3083.788286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3083.791942] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3083.827689] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3084.844488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3089.889582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3089.893467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3089.898195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3089.900167] Lustre: Skipped 1 previous similar message [ 3095.086094] LustreError: 202273:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3095.089214] LustreError: 202273:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3095.109300] Lustre: server umount lustre-OST0000 complete [ 3097.237693] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3097.323296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3098.468747] LustreError: 200678:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3098.723966] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3098.806859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3100.132536] LustreError: 200677:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.136886] LustreError: 200677:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3103.588919] LustreError: 200679:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.345244] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3109.345277] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3109.349078] LustreError: Skipped 1 previous similar message [ 3109.354646] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3109.357372] Lustre: Skipped 5 previous similar messages [ 3110.370531] LustreError: 200679:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3110.377092] LustreError: 200679:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3114.989101] LustreError: 202963:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3114.991847] LustreError: 202963:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3115.055186] Lustre: server umount lustre-OST0001 complete [ 3120.609425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3120.610098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.618052] Lustre: Skipped 2 previous similar messages [ 3122.457930] Lustre: server umount lustre-MDT0000 complete [ 3123.759296] LustreError: 199073:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012295 with bad export cookie 15808265985369995866 [ 3123.762982] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3123.764707] LustreError: 199073:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3123.834588] LustreError: 203365:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3123.838840] LustreError: 203365:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3123.935882] Lustre: server umount lustre-MDT0001 complete [ 3125.893780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3127.267807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3128.589566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3129.908213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3132.146638] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 21:38:23 (1754012303) [ 3134.314491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.402446] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3134.413092] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3134.480605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3135.724620] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3138.485739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3138.516977] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3138.534841] Lustre: 204855:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3139.898379] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3141.624296] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3142.768752] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3144.977352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3145.041955] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3146.941431] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3147.467680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3148.642953] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3150.788194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3150.813297] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3152.706598] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3154.492606] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3157.985139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3157.985286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3157.987367] LustreError: Skipped 1 previous similar message [ 3157.991884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3157.995029] Lustre: Skipped 5 previous similar messages [ 3161.390099] LustreError: 208024:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3161.392960] LustreError: 208024:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3161.414038] Lustre: server umount lustre-OST0000 complete [ 3164.642421] LustreError: 207937:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3164.646403] LustreError: 207937:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3174.884584] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3174.887475] LustreError: Skipped 1 previous similar message [ 3178.163230] Lustre: server umount lustre-OST0001 complete [ 3185.497616] Lustre: server umount lustre-MDT0000 complete [ 3186.744085] LustreError: 204845:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012358 with bad export cookie 15808265985369997252 [ 3186.746389] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3186.748284] LustreError: 204845:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3186.894448] Lustre: server umount lustre-MDT0001 complete [ 3188.647123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3189.858429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3191.048997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3192.249472] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3194.326626] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 21:39:25 (1754012365) [ 3199.180283] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 3201.847979] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3205.025656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3207.188192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3209.060656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3211.132631] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3213.242375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3213.259287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3213.333876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3213.345955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3213.378121] Lustre: lustre-MDT0000: new disk, initializing [ 3213.399391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.401047] Lustre: Skipped 3 previous similar messages [ 3213.406023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3214.658939] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3218.771539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3218.796136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3218.830151] Lustre: 212429:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3218.852181] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3218.853987] Lustre: Skipped 1 previous similar message [ 3218.889853] Lustre: lustre-MDT0001: new disk, initializing [ 3218.937691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3218.941822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3220.281343] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3223.510426] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3224.759607] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3227.181830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3227.207901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3227.307893] Lustre: lustre-OST0000: new disk, initializing [ 3227.310237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3228.598837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3228.602364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3228.639284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3229.268683] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3232.515123] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3233.933867] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3234.032315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3235.344100] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3235.431381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3238.125078] LustreError: 215333:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3238.127560] LustreError: 215333:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3238.192372] Lustre: server umount lustre-OST0000 complete [ 3244.001623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.002464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3244.005074] Lustre: Skipped 6 previous similar messages [ 3244.007279] Lustre: Skipped 11 previous similar messages [ 3250.171426] Lustre: server umount lustre-MDT0000 complete [ 3251.417204] LustreError: 215334:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012423 with bad export cookie 15808265985369998799 [ 3251.420394] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3251.421291] LustreError: 215334:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3262.001974] Lustre: server umount lustre-MDT0001 complete [ 3264.873787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3265.025795] LustreError: 216213:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3265.031478] LustreError: 216213:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 3266.398031] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3269.311425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3270.708422] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3272.509195] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3273.756803] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3276.057661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3277.992303] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3279.759524] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3281.117407] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3290.399363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 3291.712890] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3291.789339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3294.689625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3294.693019] LustreError: Skipped 2 previous similar messages [ 3298.597512] Lustre: server umount lustre-OST0000 complete [ 3300.360068] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3301.549872] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3325.253279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3325.371271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3325.373284] Lustre: Skipped 5 previous similar messages [ 3327.308148] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3329.045821] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3329.613142] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3332.588884] LustreError: 220598:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3332.591801] LustreError: 220598:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3332.651823] Lustre: server umount lustre-OST0000 complete [ 3334.946457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3334.950308] Lustre: Skipped 2 previous similar messages [ 3334.952022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3334.954009] Lustre: Skipped 6 previous similar messages [ 3340.054173] Lustre: server umount lustre-MDT0000 complete [ 3340.257965] LustreError: 216213:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3340.263332] LustreError: 216213:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3341.301386] LustreError: 216192:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012513 with bad export cookie 15808265985369999667 [ 3341.304987] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3341.305263] LustreError: 216192:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3341.451913] Lustre: server umount lustre-MDT0001 complete [ 3344.251606] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 21:41:55 (1754012515) [ 3346.409757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.933911] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3350.791872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3352.187199] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3353.965226] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3355.074139] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3357.204217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3359.077537] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3360.775141] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3362.042084] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3372.311415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3373.519254] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3373.596175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3375.750673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3375.780550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3375.842773] Lustre: lustre-OST0001: new disk, initializing [ 3375.845595] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3377.859679] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3380.909427] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3382.175423] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3383.805079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3383.809178] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3383.835339] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3384.287819] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3385.553712] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3385.625089] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3388.900768] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3388.903779] LustreError: Skipped 1 previous similar message [ 3392.314681] Lustre: server umount lustre-OST0001 complete [ 3394.262145] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3394.342657] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3395.626078] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3395.694173] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3404.258673] LustreError: 223259:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.264083] LustreError: 223259:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 9 previous similar messages [ 3419.214564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3421.162461] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3422.945766] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3423.466713] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3430.198171] Lustre: server umount lustre-OST0001 complete [ 3448.361950] Lustre: server umount lustre-OST0000 complete [ 3455.758137] Lustre: server umount lustre-MDT0000 complete [ 3456.933417] LustreError: 221653:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012628 with bad export cookie 15808265985370000584 [ 3456.938051] LustreError: 221653:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3456.938475] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3471.328169] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3471.365300] LustreError: 228165:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3471.367883] LustreError: 228165:0:(obd_class.h:479:obd_check_dev()) Skipped 37 previous similar messages [ 3471.447201] Lustre: server umount lustre-MDT0001 complete [ 3473.430556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3474.774327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3476.076501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3477.362198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3479.578321] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 21:44:10 (1754012650) [ 3481.780775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3481.886084] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3481.898058] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3481.976107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3481.978637] Lustre: Skipped 5 previous similar messages [ 3483.323040] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3486.062928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3486.092208] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3486.101683] Lustre: 229658:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3487.427181] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3489.186735] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3490.392862] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3492.644454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3492.707894] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3494.624202] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3496.303473] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3505.668700] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3505.693289] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3507.727211] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3509.539115] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3510.074996] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3511.329892] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3511.409117] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3512.681489] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3512.763769] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3513.974507] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d36871e5800.ost_server_uuid 50 [ 3514.431733] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d36871e5800.ost_server_uuid in FULL state after 0 sec [ 3514.765137] Lustre: Permanently deactivating lustre-OST0001 [ 3514.769703] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3515.536240] Lustre: Permanently reactivating lustre-OST0001 [ 3521.508463] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3521.509859] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3521.511637] LustreError: Skipped 3 previous similar messages [ 3521.514691] Lustre: Skipped 7 previous similar messages [ 3521.515239] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3521.519945] Lustre: Skipped 16 previous similar messages [ 3522.552962] Lustre: server umount lustre-OST0001 complete [ 3536.872184] LustreError: 231825:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3536.872184] LustreError: 231269:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3536.872230] LustreError: 231825:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 17 previous similar messages [ 3536.878716] LustreError: 231269:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 17 previous similar messages [ 3537.888101] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3537.957549] Lustre: server umount lustre-OST0000 complete [ 3545.299645] Lustre: server umount lustre-MDT0000 complete [ 3546.564133] LustreError: 229650:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012718 with bad export cookie 15808265985370001844 [ 3546.567072] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3546.569246] LustreError: 229650:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3546.713042] Lustre: server umount lustre-MDT0001 complete [ 3548.606317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3549.850431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3551.076414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3552.316994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3554.404279] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:45:25 (1754012725) [ 3554.731027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3556.892198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3556.989683] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3557.002317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3557.005466] Lustre: Skipped 5 previous similar messages [ 3558.302853] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3561.028989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3561.059595] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3561.069051] Lustre: 235760:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3562.439274] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3564.245183] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3565.418968] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3567.647616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3567.669464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3567.748660] Lustre: Permanently deactivating lustre-OST0000 [ 3567.754167] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3568.281761] Lustre: lustre-OST0000_UUID: llog_process_th(pid=237477/ppid=2) setting import INACTIVE by administrator request [ 3569.694269] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3571.341592] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3573.448315] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3575.067920] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3575.257637] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3576.960302] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3577.504423] Lustre: Permanently reactivating lustre-OST0000 [ 3582.455721] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3582.458022] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3582.461151] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.124@tcp (at 0@lo) [ 3582.476155] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3582.483394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3593.406757] Lustre: server umount lustre-OST0001 complete [ 3609.056107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3609.124767] Lustre: server umount lustre-OST0000 complete [ 3616.467546] Lustre: server umount lustre-MDT0000 complete [ 3617.619353] LustreError: 235751:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012789 with bad export cookie 15808265985370003496 [ 3617.622663] LustreError: 235751:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3617.622909] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3617.759984] Lustre: server umount lustre-MDT0001 complete [ 3622.059988] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 3623.138509] Key type lgssc unregistered [ 3623.282407] LNet: 240122:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3624.294359] LNet: Removed LNI 192.168.203.124@tcp [ 3624.588788] Key type .llcrypt unregistered [ 3624.590789] Key type ._llcrypt unregistered [ 3632.684399] Key type ._llcrypt registered [ 3632.685972] Key type .llcrypt registered [ 3632.722145] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3633.046860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3633.084210] alg: No test for adler32 (adler32-zlib) [ 3633.934420] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 3634.015253] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 3634.017118] LNet: Accept secure, port 988 [ 3635.592173] Key type lgssc registered [ 3635.959949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3637.075180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3639.161485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3639.167176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.315478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3641.523342] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3644.253384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3644.276368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3644.306166] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3644.314398] Lustre: 241525:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3644.317902] Lustre: Permanently deactivating lustre-MDT0001 [ 3644.322729] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3644.423371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3645.638540] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3647.434525] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3648.750979] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3650.029454] Lustre: lustre-MDT0001_UUID: llog_process_th(pid=242969/ppid=2) setting import INACTIVE by administrator request [ 3651.048451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3651.157085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3652.992504] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3654.241146] LustreError: 243145:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3654.245095] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3654.247540] LustreError: 243145:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3654.858921] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3657.208462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3657.276901] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3659.162818] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3661.055427] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3662.311146] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3665.383530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3665.383544] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3666.719109] Lustre: Permanently reactivating lustre-MDT0001 [ 3666.725766] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3666.728300] Lustre: Skipped 2 previous similar messages [ 3670.889710] Lustre: lustre-MDT0001: Client fa1e1a45-116d-4493-a249-a9dbb9d8994b (at 192.168.203.24@tcp) reconnecting [ 3673.073317] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3673.608701] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.613736] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3673.616761] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3673.621689] LustreError: 244758:0:(llog_osd.c:2254:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 3673.622262] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 192.168.203.124@tcp (at 0@lo) [ 3673.625157] LustreError: 244758:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3673.631574] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3673.632093] LustreError: 244758:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.124@tcp: cfg command failed: rc = -108 [ 3673.639617] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 3673.642561] LustreError: 241527:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3675.337106] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3675.408071] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3676.721279] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3676.800385] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3677.606191] Lustre: Permanently deactivating lustre-MDT0001 [ 3677.612458] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3677.614964] Lustre: Skipped 2 previous similar messages [ 3682.816597] Lustre: lustre-MDT0001_UUID: llog_process_th(pid=245476/ppid=2) setting import INACTIVE by administrator request [ 3688.930318] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3688.932330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3688.935569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3690.977331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3690.978126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3690.979463] Lustre: Skipped 1 previous similar message [ 3690.982747] Lustre: Skipped 2 previous similar messages [ 3693.627985] LustreError: 245611:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3693.651602] Lustre: server umount lustre-MDT0000 complete [ 3694.839921] LustreError: 241515:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012866 with bad export cookie 11703925895472908965 [ 3694.843744] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3694.846021] LustreError: 241515:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3694.907029] LustreError: 245813:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3694.909562] LustreError: 245813:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3694.985419] Lustre: server umount lustre-MDT0001 complete [ 3702.252118] LustreError: 246013:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3702.254520] LustreError: 246013:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3702.270945] Lustre: server umount lustre-OST0000 complete [ 3705.774508] Lustre: DEBUG MARKER: == conf-sanity test 50j: deactivate MDC for MDT0 ========= 21:47:57 (1754012877) [ 3706.298246] Lustre: DEBUG MARKER: conf-sanity test_50j: @@@@@@ FAIL: failed to block deactivating MDC0