[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 539052135 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003212] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010154] LSM: Security Framework initializing [ 0.011066] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.013065] *** VALIDATE selinux *** [ 0.021826] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026025] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027094] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028100] *** VALIDATE tmpfs *** [ 0.030337] *** VALIDATE proc *** [ 0.031200] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.033247] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034137] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036023] Spectre V2 : User space: Vulnerable [ 0.037006] Speculative Store Bypass: Vulnerable [ 0.040247] debug: unmapping init [mem 0xffffffffa0a59000-0xffffffffa0a60fff] [ 0.043094] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044622] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045019] ... version: 2 [ 0.046008] ... bit width: 48 [ 0.047007] ... generic registers: 4 [ 0.048010] ... value mask: 0000ffffffffffff [ 0.049010] ... max period: 00007fffffffffff [ 0.050007] ... fixed-purpose events: 3 [ 0.050923] ... event mask: 000000070000000f [ 0.051271] rcu: Hierarchical SRCU implementation. [ 0.053384] smp: Bringing up secondary CPUs ... [ 0.054516] x86: Booting SMP configuration: [ 0.055016] .... node #0, CPUs: #1 #2 #3 [ 0.059560] smp: Brought up 1 node, 4 CPUs [ 0.061019] smpboot: Max logical packages: 1 [ 0.062012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.101947] node 0 deferred pages initialised in 35ms [ 0.103088] devtmpfs: initialized [ 0.104000] x86/mm: Memory block size: 128MB [ 0.106722] gcov: version magic: 0x41383552 [ 0.109177] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.112140] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.113325] pinctrl core: initialized pinctrl subsystem [ 0.114000] [ 0.114000] ************************************************************* [ 0.115012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.117011] ** ** [ 0.119011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.120010] ** ** [ 0.123009] ** This means that this kernel is built to expose internal ** [ 0.124014] ** IOMMU data structures, which may compromise security on ** [ 0.126012] ** your system. ** [ 0.127000] ** ** [ 0.128011] ** If you see this message and you are not debugging the ** [ 0.129008] ** kernel, report this immediately to your vendor! ** [ 0.131012] ** ** [ 0.133009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.139014] ************************************************************* [ 0.145236] NET: Registered protocol family 16 [ 0.147389] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.150077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.153057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.157006] cpuidle: using governor menu [ 0.159547] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.162416] PCI: Using configuration type 1 for base access [ 0.164116] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.174209] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.175000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.179746] cryptd: max_cpu_qlen set to 1000 [ 0.183827] ACPI: Added _OSI(Module Device) [ 0.185009] ACPI: Added _OSI(Processor Device) [ 0.187123] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.190014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.195186] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.200560] ACPI: Interpreter enabled [ 0.201000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.201000] ACPI: Using IOAPIC for interrupt routing [ 0.201000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.201000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.209543] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.212027] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.213012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.216062] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.220192] acpiphp: Slot [2] registered [ 0.222089] acpiphp: Slot [3] registered [ 0.223000] acpiphp: Slot [4] registered [ 0.223000] acpiphp: Slot [5] registered [ 0.223075] acpiphp: Slot [6] registered [ 0.224000] acpiphp: Slot [7] registered [ 0.226100] acpiphp: Slot [8] registered [ 0.227087] acpiphp: Slot [9] registered [ 0.229063] acpiphp: Slot [10] registered [ 0.230149] acpiphp: Slot [11] registered [ 0.231056] acpiphp: Slot [12] registered [ 0.232059] acpiphp: Slot [13] registered [ 0.234067] acpiphp: Slot [14] registered [ 0.238131] acpiphp: Slot [15] registered [ 0.239059] acpiphp: Slot [16] registered [ 0.240081] acpiphp: Slot [17] registered [ 0.243092] acpiphp: Slot [18] registered [ 0.245073] acpiphp: Slot [19] registered [ 0.246084] acpiphp: Slot [20] registered [ 0.247068] acpiphp: Slot [21] registered [ 0.249065] acpiphp: Slot [22] registered [ 0.251066] acpiphp: Slot [23] registered [ 0.252082] acpiphp: Slot [24] registered [ 0.253059] acpiphp: Slot [25] registered [ 0.255078] acpiphp: Slot [26] registered [ 0.257095] acpiphp: Slot [27] registered [ 0.258070] acpiphp: Slot [28] registered [ 0.260062] acpiphp: Slot [29] registered [ 0.261083] acpiphp: Slot [30] registered [ 0.263091] acpiphp: Slot [31] registered [ 0.265045] PCI host bridge to bus 0000:00 [ 0.266010] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.269015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.271015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.273013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.276014] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.277013] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.278151] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.282224] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.286727] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.297010] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.302011] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.305015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.307012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.309016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.312582] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.314732] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.317036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.318000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.322013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.336014] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.342013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.347495] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.355017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.363016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.380012] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.390010] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.394010] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.397000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.407013] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.414985] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.424015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.430012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.442019] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.453738] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.458012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.464015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.480019] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.487611] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.495015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.502013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.512013] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.522883] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.529012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.535013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.548000] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.569340] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.571325] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.572371] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.573302] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.574000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.577132] iommu: Default domain type: Passthrough [ 0.578000] SCSI subsystem initialized [ 0.578000] ACPI: bus type USB registered [ 0.578000] usbcore: registered new interface driver usbfs [ 0.579082] usbcore: registered new interface driver hub [ 0.580064] usbcore: registered new device driver usb [ 0.582187] pps_core: LinuxPPS API ver. 1 registered [ 0.583007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.586124] PTP clock support registered [ 0.592058] EDAC MC: Ver: 3.0.0 [ 0.594119] PCI: Using ACPI for IRQ routing [ 0.595673] NetLabel: Initializing [ 0.597015] NetLabel: domain hash size = 128 [ 0.599008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.604170] NetLabel: unlabeled traffic allowed by default [ 0.606124] vgaarb: loaded [ 0.609007] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.611024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.620279] clocksource: Switched to clocksource kvm-clock [ 0.769711] VFS: Disk quotas dquot_6.6.0 [ 0.771506] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.773543] *** VALIDATE ramfs *** [ 0.774582] *** VALIDATE hugetlbfs *** [ 0.775757] pnp: PnP ACPI init [ 0.777776] pnp: PnP ACPI: found 6 devices [ 0.795409] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.798201] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.799857] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.801452] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.803307] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.805146] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.807226] NET: Registered protocol family 2 [ 0.809385] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.813460] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.816385] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.822258] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.825689] TCP: Hash tables configured (established 65536 bind 65536) [ 0.828919] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.831542] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.834256] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.836973] NET: Registered protocol family 1 [ 0.840742] RPC: Registered named UNIX socket transport module. [ 0.842746] RPC: Registered udp transport module. [ 0.844272] RPC: Registered tcp transport module. [ 0.845537] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.847451] NET: Registered protocol family 44 [ 0.848658] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.850374] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.852103] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.854078] PCI: CLS 0 bytes, default 64 [ 0.856183] Unpacking initramfs... [ 2.500383] debug: unmapping init [mem 0xffff9b5f3cc54000-0xffff9b5f3ffbffff] [ 2.508770] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.510979] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.516321] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.246055] Initialise system trusted keyrings [ 3.247383] Key type blacklist registered [ 3.249128] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.264658] zbud: loaded [ 3.268529] *** VALIDATE nfs *** [ 3.269640] *** VALIDATE nfs4 *** [ 3.272202] pstore: using deflate compression [ 3.277768] Platform Keyring initialized [ 3.427164] NET: Registered protocol family 38 [ 3.428568] Key type asymmetric registered [ 3.429848] Asymmetric key parser 'x509' registered [ 3.431353] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.434951] io scheduler mq-deadline registered [ 3.436471] io scheduler kyber registered [ 3.437944] io scheduler bfq registered [ 3.439641] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.442412] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.445473] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.449263] ACPI: Power Button [PWRF] [ 3.553592] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.660258] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.874099] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.971889] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.195733] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.230535] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.259948] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.265055] Non-volatile memory driver v1.3 [ 4.266365] Linux agpgart interface v0.103 [ 4.292601] virtio_blk virtio1: [vda] 133640 512-byte logical blocks (68.4 MB/65.3 MiB) [ 4.295312] vda: detected capacity change from 0 to 68423680 [ 4.315217] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.318680] vdb: detected capacity change from 0 to 1073741824 [ 4.337618] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.341136] vdc: detected capacity change from 0 to 2621440000 [ 4.356592] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.359514] vdd: detected capacity change from 0 to 2621440000 [ 4.370920] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.373764] vde: detected capacity change from 0 to 4294967296 [ 4.386960] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.389381] vdf: detected capacity change from 0 to 4294967296 [ 4.394823] libphy: Fixed MDIO Bus: probed [ 4.412362] usbcore: registered new interface driver usbserial_generic [ 4.418291] usbserial: USB Serial support registered for generic [ 4.420541] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.424927] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.426555] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.429028] mousedev: PS/2 mouse device common for all mice [ 4.431658] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.432287] rtc_cmos 00:05: RTC can wake from S4 [ 4.438782] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.444792] rtc_cmos 00:05: registered as rtc0 [ 4.444977] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.446034] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.451651] intel_pstate: CPU model not supported [ 4.454271] hid: raw HID events driver (C) Jiri Kosina [ 4.456108] usbcore: registered new interface driver usbhid [ 4.457876] usbhid: USB HID core driver [ 4.459162] drop_monitor: Initializing network drop monitor service [ 4.461597] Initializing XFRM netlink socket [ 4.463295] NET: Registered protocol family 10 [ 4.466129] Segment Routing with IPv6 [ 4.467329] NET: Registered protocol family 17 [ 4.469737] mpls_gso: MPLS GSO support [ 4.474253] RAS: Correctable Errors collector initialized. [ 4.475909] AVX version of gcm_enc/dec engaged. [ 4.477094] AES CTR mode by8 optimization enabled [ 4.578853] sched_clock: Marking stable (4578837682, 0)->(5667202540, -1088364858) [ 4.584383] registered taskstats version 1 [ 4.586821] Loading compiled-in X.509 certificates [ 4.588382] zswap: loaded using pool lzo/zbud [ 4.628138] Key type big_key registered [ 4.640345] Key type encrypted registered [ 4.641727] ima: No TPM chip found, activating TPM-bypass! [ 4.644109] ima: Allocated hash algorithm: sha1 [ 4.645692] ima: No architecture policies found [ 4.646992] evm: Initialising EVM extended attributes: [ 4.648629] evm: security.selinux [ 4.649587] evm: security.ima [ 4.650506] evm: security.capability [ 4.651562] evm: HMAC attrs: 0x1 [ 4.655353] rtc_cmos 00:05: setting system clock to 2025-12-03 19:41:54 UTC (1764790914) [ 4.661211] debug: unmapping init [mem 0xffffffffa1a03000-0xffffffffa1bfffff] [ 4.664230] debug: unmapping init [mem 0xffffffffa0782000-0xffffffffa0a58fff] [ 4.673050] Write protecting the kernel read-only data: 28672k [ 4.678241] debug: unmapping init [mem 0xffffffff9ee03000-0xffffffff9effffff] [ 4.682318] debug: unmapping init [mem 0xffffffff9f714000-0xffffffff9f7fffff] [ 4.724433] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.760301] systemd[1]: Detected virtualization kvm. [ 4.761761] systemd[1]: Detected architecture x86-64. [ 4.763519] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.808746] systemd[1]: No hostname configured. [ 4.810270] systemd[1]: Set hostname to . [ 4.811972] random: systemd: uninitialized urandom read (16 bytes read) [ 4.814160] systemd[1]: Initializing machine ID from random generator. [ 5.043507] random: systemd: uninitialized urandom read (16 bytes read) [ 5.046552] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.052485] random: systemd: uninitialized urandom read (16 bytes read) [ 5.055126] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.059455] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.216715] device-mapper: uevent: version 1.0.3 [ 6.219066] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.686729] virtio_net virtio0 ens2: renamed from eth0 [ 7.792104] random: fast init done [ 8.647912] scsi host0: ata_piix [ 8.854935] scsi host1: ata_piix [ 8.862741] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.866601] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.166516] random: crng init done [ 13.168317] random: 7 urandom warning(s) missed due to ratelimiting [ 14.439914] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 15.552625] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.877386] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.418356] SELinux: Disabled at runtime. [ 18.579668] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.596047] systemd[1]: Detected virtualization kvm. [ 18.598370] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.250263] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.259701] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.268877] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.274444] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.277663] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.292373] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.299276] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ 20.516769] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 21.718324] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.444964] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.626980] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.182286] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.365108] EDAC sbridge: Ver: 1.1.2 [ 27.038134] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 27.459638] NFS: Registering the id_resolver key type [ 27.461283] Key type id_resolver registered [ 27.462497] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg320-server login: [ 56.180501] libcfs: loading out-of-tree module taints kernel. [ 56.195595] Key type ._llcrypt registered [ 56.197149] Key type .llcrypt registered [ 56.240431] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 63.688648] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 64.311209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.319560] alg: No test for adler32 (adler32-zlib) [ 65.310509] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 65.582192] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 67.199157] Key type lgssc registered [ 67.700581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 74.260726] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 74.678590] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 78.109865] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 80.831425] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.211853] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 91.820538] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 97.824029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 97.847968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 97.857687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 98.978452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 98.996689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 99.046386] Lustre: lustre-MDT0000: new disk, initializing [ 99.095222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 99.104857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 100.777853] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 106.377795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.408699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 106.445245] Lustre: 6481:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 106.458886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 106.461819] Lustre: Skipped 1 previous similar message [ 106.501374] Lustre: lustre-MDT0001: new disk, initializing [ 106.524974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 106.537143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 106.542051] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 107.834371] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 110.039723] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 113.236233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 113.260592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 113.337911] Lustre: lustre-OST0000: new disk, initializing [ 113.339722] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 113.354018] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 115.463957] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 117.741180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 117.745258] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 117.771311] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 123.217126] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 123.323126] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 123.403645] Lustre: lustre-OST0001: new disk, initializing [ 123.406131] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 123.452443] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 126.947136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 126.951118] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 127.044240] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 128.000335] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 138.793447] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 148.203641] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 158.878478] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing check_logdir /tmp/testlogs/ [ 162.742772] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing yml_node [ 165.822950] Lustre: DEBUG MARKER: Client: 2.16.59.23 [ 167.421939] Lustre: DEBUG MARKER: MDS: 2.16.59.23 [ 168.849705] Lustre: DEBUG MARKER: OSS: 2.16.59.23 [ 169.737282] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Dec 3 14:44:38 EST 2025 [ 178.178794] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 178.809644] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 193.503965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 193.504449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 193.508265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 193.516697] Lustre: Skipped 2 previous similar messages [ 198.624352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 198.627050] Lustre: Skipped 6 previous similar messages [ 203.744791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 205.279911] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 205.330247] LustreError: 12183:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 205.368354] Lustre: server umount lustre-MDT0000 complete [ 207.065765] LustreError: 6516:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764791116 with bad export cookie 13768224973000794263 [ 207.066528] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 207.072631] LustreError: 6516:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 208.864970] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 208.865925] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 208.870362] Lustre: Skipped 2 previous similar messages [ 208.874767] Lustre: Skipped 3 previous similar messages [ 212.898050] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 212.900894] Lustre: Skipped 1 previous similar message [ 213.187593] LustreError: 12386:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 213.190454] LustreError: 12386:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 213.280311] Lustre: server umount lustre-MDT0001 complete [ 215.053131] LustreError: 12587:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 215.056568] LustreError: 12587:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 215.081762] Lustre: server umount lustre-OST0000 complete [ 217.072929] LustreError: 12787:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 217.074995] LustreError: 12787:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 217.171528] Lustre: server umount lustre-OST0001 complete [ 220.823870] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 228.920386] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 238.773624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 245.071307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 251.726265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 258.751294] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 265.874832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 265.940342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 266.205815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 266.261839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 266.356703] Lustre: lustre-MDT0000: new disk, initializing [ 266.475987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 266.503499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 270.113185] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 277.424820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 277.460914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 277.525549] Lustre: 14970:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 277.569254] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 277.571767] Lustre: Skipped 1 previous similar message [ 277.610640] Lustre: lustre-MDT0001: new disk, initializing [ 277.689966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 277.722595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 277.734374] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 280.389391] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 285.696427] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 288.099227] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 293.331869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 293.376211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 294.748037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 294.754556] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 294.854385] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 297.675507] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 302.939865] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 305.529372] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 305.688558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 308.302160] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 308.450535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 310.245342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.245395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 310.249332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 310.251798] Lustre: Skipped 1 previous similar message [ 315.705259] LustreError: 17908:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 315.709791] LustreError: 17908:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 315.815243] Lustre: server umount lustre-OST0000 complete [ 318.945783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 318.954976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.702862] LustreError: 18109:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 324.707363] LustreError: 18109:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 324.809499] Lustre: server umount lustre-MDT0000 complete [ 325.601455] LustreError: 15921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.616629] LustreError: 15921:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 327.636523] LustreError: 14964:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764791237 with bad export cookie 13768224973000795789 [ 327.638431] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 327.643756] LustreError: 14964:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 327.873930] Lustre: server umount lustre-MDT0001 complete [ 331.684432] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 14:47:19 (1764791239) [ 343.277661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 343.526490] LustreError: 19449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.575178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 343.578262] Lustre: Skipped 1 previous similar message [ 346.207927] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 348.641884] LustreError: 19449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 353.461958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 353.761650] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 353.765462] Lustre: Skipped 6 previous similar messages [ 357.182399] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 360.695904] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 362.891591] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 364.211465] Lustre: Failing over lustre-MDT0000 [ 364.315808] LustreError: 20791:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 364.317849] LustreError: 20791:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 364.353575] Lustre: server umount lustre-MDT0000 complete [ 368.609277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.620223] Lustre: Skipped 1 previous similar message [ 368.631389] LustreError: 19445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 368.639311] LustreError: 19445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 373.730162] LustreError: 19449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.742685] LustreError: 19449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 380.112817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.181964] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 380.323832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 380.326494] Lustre: Skipped 1 previous similar message [ 382.308810] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 385.505682] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 385.508298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 385.520652] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 386.576129] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 389.010495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 389.011373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 389.015254] Lustre: Skipped 2 previous similar messages [ 395.230591] Lustre: server umount lustre-MDT0000 complete [ 397.253501] LustreError: 20792:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764791307 with bad export cookie 13768224973000797210 [ 397.255990] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 397.260588] LustreError: 20792:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 397.378848] LustreError: 21972:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 397.382170] LustreError: 21972:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 397.512723] Lustre: server umount lustre-MDT0001 complete [ 402.303419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 402.524971] LustreError: 22448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 402.534533] LustreError: 22448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 404.526397] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 408.785712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 411.409684] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 415.245844] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 418.720809] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 420.653396] hrtimer: interrupt took 9387058 ns [ 421.283857] Lustre: Failing over lustre-MDT0000 [ 421.310486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 421.864746] Lustre: server umount lustre-MDT0000 complete [ 426.466838] LustreError: 22447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 426.525592] LustreError: 22447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 442.853089] Lustre: 3628:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764791336/real 1764791336] req@ffff9b5fbfc4d500 x1850517461561984/t0(0) o400->MGC192.168.203.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1764791352 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 442.906293] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 445.837278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 453.092814] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbf128faa6f5fefbd [ 453.108291] Lustre: MGC192.168.203.120@tcp: Connection restored to 0@lo (at 0@lo) [ 453.111344] Lustre: Skipped 1 previous similar message [ 453.591539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 453.594525] Lustre: Skipped 2 previous similar messages [ 454.568224] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 454.574171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 454.622754] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 457.657433] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 465.491079] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 469.984301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 469.987463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.020079] Lustre: Skipped 1 previous similar message [ 470.031890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 470.033872] Lustre: Skipped 5 previous similar messages [ 483.295125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 483.376185] LustreError: 24779:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 483.380129] LustreError: 24779:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 483.430401] Lustre: server umount lustre-MDT0000 complete [ 484.321758] LustreError: 22448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 484.346564] LustreError: 22448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 486.153284] LustreError: 22428:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764791395 with bad export cookie 13768224973000798141 [ 486.155443] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 486.158649] LustreError: 22428:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 486.426517] Lustre: server umount lustre-MDT0001 complete [ 492.323717] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 14:50:00 (1764791400) [ 501.353762] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 505.875346] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 510.760552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 513.581292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 516.235769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 519.256818] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 524.607451] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 530.633468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 530.681383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.815565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 530.838119] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 530.841207] Lustre: Skipped 1 previous similar message [ 530.891186] Lustre: lustre-MDT0000: new disk, initializing [ 530.893344] Lustre: Skipped 1 previous similar message [ 530.953893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 533.060442] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 538.951033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 538.995502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 539.050775] Lustre: 28444:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 539.188924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 540.979680] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 544.767713] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 546.284641] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 549.490041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 549.524914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 551.335895] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 552.224538] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 555.856831] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 556.210893] Lustre: 28450:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 556.215342] Lustre: 28450:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 556.221522] Lustre: 28450:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 556.228106] Lustre: 28450:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/153/0, punch: 0/0/0, quota 1/3/0 [ 556.233849] Lustre: 28450:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 556.236326] Lustre: 28450:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 556.521703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 556.530709] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 556.532460] Lustre: Skipped 1 previous similar message [ 556.534342] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 559.660683] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 566.753525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.754876] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 566.757582] Lustre: Skipped 1 previous similar message [ 572.807292] Lustre: server umount lustre-OST0000 complete [ 574.943855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 574.945962] LustreError: Skipped 1 previous similar message [ 580.342812] Lustre: server umount lustre-MDT0000 complete [ 582.004919] LustreError: 28436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764791491 with bad export cookie 13768224973000798547 [ 582.008159] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 582.013169] LustreError: 28436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 582.272631] Lustre: server umount lustre-MDT0001 complete [ 586.839208] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 588.030996] Key type lgssc unregistered [ 588.168473] LNet: 32013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.171396] LNetError: 32013:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 588.181370] LNet: Removed LNI 192.168.203.120@tcp [ 588.583337] Key type .llcrypt unregistered [ 588.584899] Key type ._llcrypt unregistered [ 599.886303] Key type ._llcrypt registered [ 599.887746] Key type .llcrypt registered [ 599.933580] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 600.296332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 600.340075] alg: No test for adler32 (adler32-zlib) [ 601.220262] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 601.323938] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 602.935240] Key type lgssc registered [ 603.351324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 606.589473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 606.598207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.841408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 609.494877] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 612.897340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 613.035652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 614.516238] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 616.480761] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 617.938587] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 620.606525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 620.784317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 622.827497] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 623.149265] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 625.171541] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 640.277800] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 14:52:29 (1764791549) [ 640.942091] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 641.572454] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 643.722022] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 14:52:32 (1764791552) [ 646.531788] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 14:52:35 (1764791555) [ 646.874224] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 649.590202] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 649.593247] Lustre: Skipped 2 previous similar messages [ 650.126447] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 14:52:38 (1764791558) [ 657.023187] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 657.026673] Lustre: Skipped 10 previous similar messages [ 657.950084] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 14:52:46 (1764791566) [ 658.944514] Lustre: *** cfs_fail_loc=131b, val=2*** [ 660.218939] Lustre: *** cfs_fail_loc=131b, val=2*** [ 663.802964] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 14:52:52 (1764791572) [ 665.738477] Lustre: Disabling parameter general.jobid_name= in log params [ 665.741020] Lustre: Skipped 4 previous similar messages [ 675.884601] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 14:53:03 (1764791583) [ 689.637278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 689.638939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.652478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 689.674168] Lustre: Skipped 1 previous similar message [ 691.281361] LustreError: 39948:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 691.444984] Lustre: server umount lustre-OST0000 complete [ 694.754701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.771097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 694.773976] Lustre: Skipped 1 previous similar message [ 699.875336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 699.882116] Lustre: Skipped 1 previous similar message [ 704.995235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 705.002305] Lustre: Skipped 1 previous similar message [ 709.087217] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 709.198019] LustreError: 40149:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 709.204373] LustreError: 40149:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 709.287413] Lustre: server umount lustre-MDT0000 complete [ 710.118070] LustreError: 34641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 710.143604] LustreError: 34641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 712.241384] LustreError: 35989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764791622 with bad export cookie 12095306319757661657 [ 712.246861] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 712.251995] LustreError: 35989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 712.381214] LustreError: 40351:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 712.387578] LustreError: 40351:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 712.708095] Lustre: server umount lustre-MDT0001 complete [ 723.015548] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 725.895889] Key type lgssc unregistered [ 726.166615] LNet: 40883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.176991] LNetError: 40883:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 726.203541] LNet: Removed LNI 192.168.203.120@tcp [ 726.957669] Key type .llcrypt unregistered [ 726.958983] Key type ._llcrypt unregistered [ 737.254934] Key type ._llcrypt registered [ 737.256499] Key type .llcrypt registered [ 737.322947] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 745.703888] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 746.273753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 746.281296] alg: No test for adler32 (adler32-zlib) [ 747.209701] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 747.320493] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 748.935194] Key type lgssc registered [ 749.598549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 753.622354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 756.790695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 759.627408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 762.845645] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 767.240554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 767.288712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 767.306343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.508732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 768.540093] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 768.608328] Lustre: lustre-MDT0000: new disk, initializing [ 768.725168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 768.746591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 773.126567] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 785.809713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 785.885153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 785.986524] Lustre: 43986:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 786.034466] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 786.065365] Lustre: Skipped 1 previous similar message [ 786.193524] Lustre: lustre-MDT0001: new disk, initializing [ 786.304542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 786.357768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 786.364725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 790.388422] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 798.016289] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 802.142260] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 809.436318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 809.549714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 809.890152] Lustre: lustre-OST0000: new disk, initializing [ 809.901247] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 810.011444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 816.352992] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 817.252995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 817.255975] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 817.421173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 823.763390] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 827.283053] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 827.394325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 830.841500] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 831.006126] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 832.993582] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 832.994366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 833.008112] LustreError: Skipped 1 previous similar message [ 833.012934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 833.027477] Lustre: Skipped 1 previous similar message [ 838.119431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 838.123705] Lustre: Skipped 2 previous similar messages [ 843.234138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 843.237968] Lustre: Skipped 1 previous similar message [ 847.327403] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 847.399161] LustreError: 46925:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 847.632950] Lustre: server umount lustre-OST0000 complete [ 852.973638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 852.989145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 853.014679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 858.593324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 858.595178] Lustre: Skipped 2 previous similar messages [ 866.271164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 866.417810] LustreError: 47128:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 866.428276] LustreError: 47128:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 866.599034] Lustre: server umount lustre-MDT0000 complete [ 873.025513] LustreError: 43978:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764791782 with bad export cookie 10239184680169295557 [ 873.046830] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 873.066175] LustreError: 43978:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 873.330895] LustreError: 47331:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 873.342448] LustreError: 47331:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 873.837956] Lustre: server umount lustre-MDT0001 complete [ 883.202819] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 14:56:30 (1764791790) [ 888.743132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 889.133305] LustreError: 48051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 889.143206] LustreError: 48051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 889.213416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 892.878292] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 894.434701] LustreError: 48051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 898.528433] LustreError: 48051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 901.190838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 901.462692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 904.799981] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 909.557565] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 913.464116] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 919.878964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 920.170553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 926.160572] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 932.002689] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 940.314476] LustreError: 50249:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 941.234453] Lustre: Modifying parameter general.timeout=1 in log params [ 943.094024] Lustre: Modifying parameter general.timeout=3 in log params [ 943.095945] Lustre: Skipped 1 previous similar message [ 945.689695] Lustre: Modifying parameter general.timeout=6 in log params [ 945.692857] Lustre: Skipped 2 previous similar messages [ 949.885261] Lustre: Modifying parameter general.timeout=11 in log params [ 949.889400] Lustre: Skipped 4 previous similar messages [ 958.529065] Lustre: Modifying parameter general.timeout=21 in log params [ 958.532646] Lustre: Skipped 9 previous similar messages [ 974.801255] Lustre: Modifying parameter general.timeout=40 in log params [ 974.803417] Lustre: Skipped 18 previous similar messages [ 1006.939102] Lustre: Modifying parameter general.timeout=81 in log params [ 1006.946017] Lustre: Skipped 40 previous similar messages [ 1055.502771] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 14:59:23 (1764791963) [ 1057.610829] Lustre: 57187:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1059.293524] Lustre: 57286:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1062.117405] Lustre: 57496:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1062.131819] Lustre: 57496:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 1 previous similar message [ 1071.382716] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1071.386543] Lustre: Skipped 60 previous similar messages [ 1093.684321] Lustre: 59051:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1093.713395] Lustre: 59051:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 2 previous similar messages [ 1093.746831] LustreError: 59051:0:(mgs_handler.c:1126:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1099.544595] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 15:00:06 (1764792006) [ 1116.641477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1116.649120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.656770] Lustre: Skipped 1 previous similar message [ 1116.660075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1116.662804] Lustre: Skipped 3 previous similar messages [ 1120.236600] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1120.292799] Lustre: Skipped 1 previous similar message [ 1122.815765] LustreError: 59753:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1122.828866] LustreError: 59753:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1123.093885] Lustre: server umount lustre-MDT0000 complete [ 1125.347197] LustreError: 48050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.369810] LustreError: 48050:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1128.583220] LustreError: 57089:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764792038 with bad export cookie 10239184680169296446 [ 1128.584050] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1128.648464] LustreError: 57089:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1128.885187] LustreError: 59954:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1128.887958] LustreError: 59954:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1129.225726] Lustre: server umount lustre-MDT0001 complete [ 1145.323110] LustreError: 60155:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1145.341237] LustreError: 60155:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1145.628435] Lustre: server umount lustre-OST0000 complete [ 1154.005396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1158.553771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1163.087455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1167.751375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1176.499172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1176.909774] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1177.185415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.483354] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1190.262982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1190.327042] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1190.356455] Lustre: 61621:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1193.398358] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1197.192529] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1199.617162] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1204.668686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1204.792180] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1209.429568] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1216.176426] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1217.932687] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1217.952449] Lustre: Skipped 11 previous similar messages [ 1218.017757] Lustre: 63854:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1226.210437] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1226.213886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1226.221080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1226.224978] Lustre: Skipped 3 previous similar messages [ 1231.094105] LustreError: 64101:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1231.105871] LustreError: 64101:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1231.279230] Lustre: server umount lustre-OST0000 complete [ 1236.960667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.960895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1236.967677] Lustre: Skipped 1 previous similar message [ 1236.970202] LustreError: Skipped 1 previous similar message [ 1240.635441] Lustre: server umount lustre-MDT0000 complete [ 1242.091712] LustreError: 62355:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1242.123970] LustreError: 62355:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1243.629885] LustreError: 61614:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764792153 with bad export cookie 10239184680169299393 [ 1243.654064] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1243.927858] Lustre: server umount lustre-MDT0001 complete [ 1251.411992] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 1253.202653] Key type lgssc unregistered [ 1253.382498] LNet: 65042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.390339] LNetError: 65042:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1254.437831] LNet: Removed LNI 192.168.203.120@tcp [ 1255.166675] Key type .llcrypt unregistered [ 1255.170186] Key type ._llcrypt unregistered [ 1272.939672] Key type ._llcrypt registered [ 1272.942170] Key type .llcrypt registered [ 1273.043770] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1273.691272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1273.718653] alg: No test for adler32 (adler32-zlib) [ 1274.765505] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 1274.960145] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 1276.599837] Key type lgssc registered [ 1277.707318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1284.383543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1284.422402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1285.951318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1289.296672] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1295.337914] LustreError: 66417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.368322] LustreError: 66417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1297.130874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1297.663720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1300.715814] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1305.420433] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1308.346482] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1313.478618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1313.882768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.760993] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1323.171247] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1339.977062] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1348.576666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1348.583796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1348.612315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1351.142180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1351.147495] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1352.292604] LustreError: 69152:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1352.379167] Lustre: server umount lustre-MDT0000 complete [ 1355.209719] LustreError: 68793:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764792265 with bad export cookie 16050077035270059283 [ 1355.218163] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1355.309777] LustreError: 69353:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1355.312204] LustreError: 69353:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1355.454575] Lustre: server umount lustre-MDT0001 complete [ 1368.636808] LustreError: 69554:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1368.641830] LustreError: 69554:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1368.801247] Lustre: server umount lustre-OST0000 complete [ 1376.078687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1379.306583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1382.270842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1385.614207] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1391.507118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1391.803244] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1391.853833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1392.192118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.180893] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1403.953333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1404.101486] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1404.127548] Lustre: 71005:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1404.722021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1409.651776] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1418.701698] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1424.717539] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1435.324277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1435.588132] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1435.999312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1443.613339] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1449.558520] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1451.412966] Lustre: 73247:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1453.859710] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1460.195719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1460.201164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1460.203839] Lustre: Skipped 1 previous similar message [ 1460.205122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1460.218274] Lustre: Skipped 3 previous similar messages [ 1465.335584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1465.349367] Lustre: Skipped 1 previous similar message [ 1466.314176] LustreError: 73513:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1466.324902] LustreError: 73513:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1466.805432] Lustre: server umount lustre-OST0000 complete [ 1475.555027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.560338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1475.588586] Lustre: Skipped 2 previous similar messages [ 1475.632799] Lustre: Skipped 1 previous similar message [ 1478.885101] LustreError: 73715:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1478.904562] LustreError: 73715:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1479.175736] Lustre: server umount lustre-MDT0000 complete [ 1480.678179] LustreError: 73423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1480.735601] LustreError: 73423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1484.230809] LustreError: 73251:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764792394 with bad export cookie 16050077035270060130 [ 1484.241119] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1484.252754] LustreError: 73251:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1485.128732] Lustre: server umount lustre-MDT0001 complete [ 1498.472506] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 1501.441285] Key type lgssc unregistered [ 1501.812280] LNet: 74450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.819536] LNetError: 74450:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1502.886794] LNet: Removed LNI 192.168.203.120@tcp [ 1503.538284] Key type .llcrypt unregistered [ 1503.540973] Key type ._llcrypt unregistered [ 1525.431746] Key type ._llcrypt registered [ 1525.433932] Key type .llcrypt registered [ 1525.507710] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1526.554489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1526.584123] alg: No test for adler32 (adler32-zlib) [ 1527.808171] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 1528.247626] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 1530.039165] Key type lgssc registered [ 1531.463811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1539.304929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1539.347039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1541.035740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1545.435796] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1550.320601] LustreError: 75844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1550.355435] LustreError: 75844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1553.908488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1554.674645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1558.506289] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1563.810854] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1567.279936] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1573.603713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1574.054794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1579.690011] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1585.374184] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1595.678627] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1596.566045] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1598.403146] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1598.414635] Lustre: Skipped 1 previous similar message [ 1601.336209] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1601.338158] Lustre: Skipped 2 previous similar messages [ 1605.890599] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1605.893904] Lustre: Skipped 4 previous similar messages [ 1614.189691] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1614.199640] Lustre: Skipped 9 previous similar messages [ 1630.223179] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1630.225976] Lustre: Skipped 21 previous similar messages [ 1662.546104] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1662.554694] Lustre: Skipped 45 previous similar messages [ 1677.952730] Lustre: 83612:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1680.114120] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 15:09:48 (1764792588) [ 1681.297343] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1682.906614] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 15:09:50 (1764792590) [ 1688.032336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1688.036536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1688.051404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1689.058071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1689.058071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1689.058094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1694.182842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1694.188945] Lustre: Skipped 2 previous similar messages [ 1699.297880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1700.324923] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1700.365084] LustreError: 83909:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1700.522677] Lustre: server umount lustre-OST0000 complete [ 1704.416885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1704.418636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1704.430526] Lustre: Skipped 1 previous similar message [ 1704.440743] Lustre: Skipped 2 previous similar messages [ 1709.775392] LustreError: 84112:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1709.778268] LustreError: 84112:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1709.854329] Lustre: server umount lustre-MDT0000 complete [ 1713.406580] LustreError: 79593:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764792623 with bad export cookie 15692267631266478372 [ 1713.412383] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1713.413507] LustreError: 79593:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1713.602868] LustreError: 84313:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1713.608634] LustreError: 84313:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1713.938912] Lustre: server umount lustre-MDT0001 complete [ 1722.325483] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 1724.385986] Key type lgssc unregistered [ 1724.625223] LNet: 84845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.631638] LNetError: 84845:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1725.676555] LNet: Removed LNI 192.168.203.120@tcp [ 1726.514160] Key type .llcrypt unregistered [ 1726.517320] Key type ._llcrypt unregistered [ 1730.449229] Key type ._llcrypt registered [ 1730.450525] Key type .llcrypt registered [ 1730.551248] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1734.656906] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules [ 1735.356740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1735.579459] alg: No test for adler32 (adler32-zlib) [ 1736.888508] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 1737.146891] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 1738.928385] Key type lgssc registered [ 1741.104507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1743.104516] LustreError: 86017:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1761.019706] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1765.039366] LustreError: 86017:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1772.087987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1772.113973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1773.733822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1777.994239] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1783.265448] LustreError: 87016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1783.293302] LustreError: 87016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1787.294141] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 15:11:35 (1764792695) [ 1788.391584] LustreError: 87016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1791.166903] LustreError: 87047:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1791.173567] LustreError: 87047:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1791.178362] LustreError: 87047:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 1791.254346] LustreError: 87821:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1791.444799] Lustre: server umount lustre-MDT0000 complete [ 1803.360533] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 1805.839288] Key type lgssc unregistered [ 1806.104930] LNet: 88507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.110924] LNetError: 88507:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.123986] LNet: Removed LNI 192.168.203.120@tcp [ 1806.783777] Key type .llcrypt unregistered [ 1806.785433] Key type ._llcrypt unregistered [ 1822.901973] Key type ._llcrypt registered [ 1822.929929] Key type .llcrypt registered [ 1823.080651] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1824.264897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1824.333186] alg: No test for adler32 (adler32-zlib) [ 1825.464936] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 1825.859375] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 1827.573898] Key type lgssc registered [ 1829.334876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1838.975964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1839.010431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1840.842475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.178569] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1850.353967] LustreError: 89706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1850.412128] LustreError: 89706:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1854.938980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1855.831866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1860.189936] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1867.152597] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1871.078575] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1877.818533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1878.577739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1883.967822] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1888.347925] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1927.075408] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 15:13:54 (1764792834) [ 1928.681213] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1930.156386] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 15:13:58 (1764792838) [ 1936.873264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1936.875386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1936.890596] Lustre: Skipped 1 previous similar message [ 1939.351074] LustreError: 92617:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1939.434949] Lustre: server umount lustre-MDT0000 complete [ 1941.990194] LustreError: 89706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1942.000031] LustreError: 89706:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1942.193234] LustreError: 89687:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764792852 with bad export cookie 13374808828829948010 [ 1942.197524] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1942.199879] LustreError: 89687:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1942.291285] LustreError: 92820:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1942.294377] LustreError: 92820:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1942.472559] Lustre: server umount lustre-MDT0001 complete [ 1955.590684] LustreError: 93020:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1955.594638] LustreError: 93020:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1955.894892] Lustre: server umount lustre-OST0000 complete [ 1963.351401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1963.656457] LustreError: 93650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1963.797352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1966.366099] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1969.121720] LustreError: 93650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1973.223057] LustreError: 93650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1973.482778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1974.156325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1977.952693] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1983.113354] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1986.031362] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1990.687090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1995.686287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1995.731094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1995.841804] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1995.845506] LustreError: 93638:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1995.863277] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1995.867845] LustreError: 95513:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 1995.871868] LustreError: 95513:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1995.876286] LustreError: 95513:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1995.987982] Lustre: server umount lustre-OST0000 complete [ 1995.994767] LustreError: 95513:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2002.881860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2003.070647] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2003.074303] LustreError: 93640:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2003.089152] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2003.094942] LustreError: 95978:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2003.098337] LustreError: 95978:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2003.106780] LustreError: 95978:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2003.220283] Lustre: server umount lustre-OST0000 complete [ 2003.229953] LustreError: 95978:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2006.265181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2012.550094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2012.678738] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2012.681300] Lustre: Found index 0 for lustre-OST0000, updating log [ 2012.685212] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2012.742122] Lustre: lustre-OST0000: new disk, initializing [ 2012.744969] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2012.968202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2018.305829] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2021.432264] Lustre: Failing over lustre-OST0000 [ 2021.465521] LustreError: 97066:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2021.471639] LustreError: 97066:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2021.624560] Lustre: server umount lustre-OST0000 complete [ 2024.929818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2024.934147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2024.938210] Lustre: Skipped 1 previous similar message [ 2024.940358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2024.942722] Lustre: Skipped 2 previous similar messages [ 2029.538469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2029.540739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2034.656643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2034.659377] Lustre: Skipped 2 previous similar messages [ 2039.264387] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2039.343216] LustreError: 97268:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2039.345872] LustreError: 97268:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2039.419840] Lustre: server umount lustre-MDT0000 complete [ 2039.776152] LustreError: 93651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2039.789328] LustreError: 93651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2040.803963] Lustre: 89016:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764792934/real 1764792934] req@0000000003ee7f66 x1850519306984960/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1764792950 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2040.836962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.643746] LustreError: 97067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764792953 with bad export cookie 13374808828829950754 [ 2043.652334] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2043.660725] LustreError: 97067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2044.112957] Lustre: server umount lustre-MDT0001 complete [ 2053.502608] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 15:16:01 (1764792961) [ 2069.971820] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 2081.482231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.815880] LustreError: 98840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2081.831293] LustreError: 98840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2081.917278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2085.324810] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2093.560509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2098.719768] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2102.673457] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2109.391508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2109.925285] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2115.481245] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2119.139293] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2119.170059] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2119.170438] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2125.208808] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2125.321751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2125.481323] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2125.641259] Lustre: lustre-OST0001: new disk, initializing [ 2125.658110] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2126.032485] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2126.043175] Lustre: Skipped 2 previous similar messages [ 2131.599390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2131.612968] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2131.765163] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2131.782596] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2139.994804] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2145.107219] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2155.107457] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.20@tcp (stopping) [ 2155.488472] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2155.491817] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2165.318316] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.20@tcp (stopping) [ 2165.321882] Lustre: Skipped 8 previous similar messages [ 2168.287143] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2168.386424] LustreError: 102081:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2168.389017] LustreError: 102081:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2168.512215] Lustre: server umount lustre-MDT0001 complete [ 2170.450627] LustreError: 98835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.459419] LustreError: 98835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2171.128640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2177.347245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2177.448850] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2177.451745] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2177.510715] Lustre: 98829:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2179.574686] LustreError: 102698:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2179.581818] LustreError: 102698:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2179.590428] LustreError: 98831:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2179.711094] LustreError: 102708:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.203.120@tcp: cfg command failed: rc = -17 [ 2179.716636] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2179.716636] [ 2181.617704] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2183.679555] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2192.122186] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 15:18:20 (1764793100) [ 2210.032202] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 2231.839340] Lustre: 104693:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2269.135613] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2275.387626] Lustre: 105830:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2280.976659] Lustre: 98836:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 506 < left 1007, rollback = 2 [ 2280.988183] Lustre: 98836:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 4/16/6, destroy: 0/0/0 [ 2280.999157] Lustre: 98836:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 8/1007/0 [ 2281.013334] Lustre: 98836:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/153/0, punch: 0/0/0, quota 1/3/0 [ 2281.037990] Lustre: 98836:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 12/236/4, delete: 0/0/0 [ 2281.057964] Lustre: 98836:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0 [ 2349.087461] Lustre: 101126:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 509 < left 993, rollback = 2 [ 2349.094673] Lustre: 101126:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 8 previous similar messages [ 2349.105549] Lustre: 101126:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/4, destroy: 0/0/0 [ 2349.115474] Lustre: 101126:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2349.144723] Lustre: 101126:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 7/993/0 [ 2349.148621] Lustre: 101126:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2349.162340] Lustre: 101126:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 5/58/0, punch: 0/0/0, quota 1/3/0 [ 2349.173934] Lustre: 101126:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2349.188947] Lustre: 101126:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 10/203/3, delete: 0/0/0 [ 2349.200853] Lustre: 101126:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2349.219975] Lustre: 101126:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0 [ 2349.232678] Lustre: 101126:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2393.055732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2393.059817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2393.065931] Lustre: Skipped 2 previous similar messages [ 2393.068206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2393.072177] Lustre: Skipped 3 previous similar messages [ 2396.581808] LustreError: 106315:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2396.584652] LustreError: 106315:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2396.636666] Lustre: server umount lustre-MDT0000 complete [ 2398.690652] LustreError: 98835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.702181] LustreError: 98835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2398.742707] LustreError: 98821:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764793308 with bad export cookie 13374808828829951755 [ 2398.744823] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2398.752547] LustreError: 98821:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2398.970246] Lustre: server umount lustre-MDT0001 complete [ 2412.099659] Lustre: server umount lustre-OST0000 complete [ 2424.646443] Lustre: server umount lustre-OST0001 complete [ 2427.827294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2432.506674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2433.083694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2437.847588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2442.547552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2443.149416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2452.382580] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 2457.571823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2457.582804] Lustre: lustre-MDT0000: reset Object Index mappings [ 2457.884276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2457.887899] Lustre: Skipped 1 previous similar message [ 2459.993584] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2464.267289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2464.274535] Lustre: lustre-MDT0001: reset Object Index mappings [ 2466.597616] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2468.075930] Lustre: 110905:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2471.130419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2473.823567] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2474.535834] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:231 to 0x280000400:257) [ 2477.815881] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2479.930384] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2481.063188] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2482.996362] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2483.003360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 2491.693761] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2494.113421] Lustre: 112760:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2530.900452] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 15:23:59 (1764793439) [ 2559.971132] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2559.973208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2559.977983] Lustre: Skipped 6 previous similar messages [ 2559.981928] Lustre: Skipped 5 previous similar messages [ 2565.349597] LustreError: 113113:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2565.352055] LustreError: 113113:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2565.391914] Lustre: server umount lustre-MDT0000 complete [ 2567.867804] LustreError: 109773:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764793477 with bad export cookie 13374808828829998620 [ 2567.878042] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2567.880400] LustreError: 109773:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2570.208588] LustreError: 109787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.218719] LustreError: 109787:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2574.290879] Lustre: server umount lustre-MDT0001 complete [ 2583.150444] Lustre: server umount lustre-OST0000 complete [ 2585.490617] LustreError: 113718:0:(ldlm_resource.c:981:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000400:0xbf:0x0].0x0 (ffff9b5fc003bf00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2591.796617] Lustre: server umount lustre-OST0001 complete [ 2596.058711] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 2601.989590] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 2607.857846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2612.072093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2615.505872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2619.163108] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2623.704987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2623.748938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.896053] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2623.928993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2623.974410] Lustre: lustre-MDT0000: new disk, initializing [ 2624.043629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2624.048190] Lustre: Skipped 3 previous similar messages [ 2624.065395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2626.672844] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2631.617076] LustreError: 116468:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2631.619582] LustreError: 116468:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2631.786669] Lustre: server umount lustre-MDT0000 complete [ 2633.593456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2637.432743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2637.476275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2637.649083] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2637.659654] Lustre: 116988:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2637.668063] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2639.654496] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2642.885645] Lustre: server umount lustre-MDT0000 complete [ 2647.962686] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 15:25:56 (1764793556) [ 2648.874024] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2649.944704] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 15:25:58 (1764793558) [ 2659.850617] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 2664.566819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2666.747525] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2671.316884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2671.364479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2671.423360] Lustre: 119001:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2671.460965] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2671.463533] Lustre: Skipped 1 previous similar message [ 2671.504511] Lustre: lustre-MDT0001: new disk, initializing [ 2671.582773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2671.589309] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2673.569851] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2676.577908] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2680.294836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2680.338217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2680.489768] Lustre: lustre-OST0000: new disk, initializing [ 2680.492820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2682.336666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2682.343433] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2682.414982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2683.406241] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2689.426162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2689.469248] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2689.552039] Lustre: lustre-OST0001: new disk, initializing [ 2689.555068] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2690.847778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2690.851810] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2690.909942] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2692.594928] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2698.968619] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2701.331975] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2706.509135] Lustre: 119009:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 506 < left 1021, rollback = 2 [ 2706.520595] Lustre: 119009:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 8 previous similar messages [ 2706.524610] Lustre: 119009:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 4/16/6, destroy: 0/0/0 [ 2706.528046] Lustre: 119009:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2706.531598] Lustre: 119009:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 9/1021/0 [ 2706.536743] Lustre: 119009:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2706.541244] Lustre: 119009:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/153/0, punch: 0/0/0, quota 1/3/0 [ 2706.545987] Lustre: 119009:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2706.550097] Lustre: 119009:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 13/232/4, delete: 0/0/0 [ 2706.553506] Lustre: 119009:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2706.558818] Lustre: 119009:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 5/5/0, ref_del: 0/0/0 [ 2706.563045] Lustre: 119009:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2930.890277] Lustre: 127774:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2930.894570] Lustre: 127774:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 8 previous similar messages [ 2930.898022] Lustre: 127774:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2930.903263] Lustre: 127774:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2930.906733] Lustre: 127774:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2930.911091] Lustre: 127774:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2930.915597] Lustre: 127774:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2930.920400] Lustre: 127774:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2930.924682] Lustre: 127774:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2930.928137] Lustre: 127774:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2930.931786] Lustre: 127774:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2930.935286] Lustre: 127774:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 3080.977098] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:33:09 (1764793989) [ 3093.474860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3093.475664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3093.482259] Lustre: Skipped 4 previous similar messages [ 3093.485174] Lustre: Skipped 10 previous similar messages [ 3099.470472] LustreError: 127991:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3099.472938] LustreError: 127991:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3099.554088] Lustre: server umount lustre-MDT0000 complete [ 3101.203411] LustreError: 127786:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764794011 with bad export cookie 13374808828830061060 [ 3101.206462] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3101.207468] LustreError: 127786:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3103.713240] LustreError: 122205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3103.720441] LustreError: 122205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3107.498326] Lustre: server umount lustre-MDT0001 complete [ 3115.403899] Lustre: server umount lustre-OST0000 complete [ 3131.359181] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3131.488714] Lustre: server umount lustre-OST0001 complete [ 3134.365340] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 3138.261300] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 3142.892115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3145.850588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3148.985602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3152.025275] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3155.322548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3155.352259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3155.474224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3155.492548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3155.539119] Lustre: lustre-MDT0000: new disk, initializing [ 3155.584714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3155.588020] Lustre: Skipped 5 previous similar messages [ 3155.601020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3157.452578] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3162.605761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3162.637229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3162.683806] Lustre: 130778:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3162.812509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3164.514545] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3168.197935] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3169.862226] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3173.064548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3173.096405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3174.602041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3174.606494] Lustre: Skipped 1 previous similar message [ 3174.613219] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3174.672096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3176.001553] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3179.817358] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3184.072888] Lustre: lustre-MDD0000: changelog on [ 3185.283813] Lustre: lustre-MDD0001: changelog on [ 3186.859304] Lustre: 131346:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 508 < left 539, rollback = 2 [ 3186.863257] Lustre: 131346:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 255 previous similar messages [ 3186.866530] Lustre: 131346:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 3186.869552] Lustre: 131346:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 3186.872812] Lustre: 131346:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 9/539/0 [ 3186.876151] Lustre: 131346:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 3186.879334] Lustre: 131346:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 10/93/1, punch: 0/0/0, quota 1/3/0 [ 3186.882329] Lustre: 131346:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 3186.885719] Lustre: 131346:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 4/67/1, delete: 0/0/0 [ 3186.889184] Lustre: 131346:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 3186.892717] Lustre: 131346:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3186.896013] Lustre: 131346:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 3194.867041] Lustre: 130786:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 525, rollback = 2 [ 3194.871093] Lustre: 130786:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 3455 previous similar messages [ 3194.874150] Lustre: 130786:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0 [ 3194.877050] Lustre: 130786:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 3455 previous similar messages [ 3194.880535] Lustre: 130786:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 3194.884136] Lustre: 130786:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 3455 previous similar messages [ 3194.887551] Lustre: 130786:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/39/0, punch: 0/0/0, quota 1/3/0 [ 3194.890956] Lustre: 130786:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 3455 previous similar messages [ 3194.894738] Lustre: 130786:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 3/50/0, delete: 0/0/0 [ 3194.897314] Lustre: 130786:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 3455 previous similar messages [ 3194.900960] Lustre: 130786:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3194.906801] Lustre: 130786:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3455 previous similar messages [ 3210.872104] Lustre: 131346:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 525, rollback = 2 [ 3210.879139] Lustre: 131346:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 5477 previous similar messages [ 3210.885243] Lustre: 131346:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0 [ 3210.893895] Lustre: 131346:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 5477 previous similar messages [ 3210.900172] Lustre: 131346:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 3210.905743] Lustre: 131346:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 5477 previous similar messages [ 3210.909652] Lustre: 131346:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/39/0, punch: 0/0/0, quota 1/3/0 [ 3210.917216] Lustre: 131346:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 5477 previous similar messages [ 3210.921766] Lustre: 131346:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 3/50/0, delete: 0/0/0 [ 3210.926672] Lustre: 131346:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 5477 previous similar messages [ 3210.931418] Lustre: 131346:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3210.936341] Lustre: 131346:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 5477 previous similar messages [ 3267.765635] Lustre: 133855:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 525, rollback = 2 [ 3267.770488] Lustre: 133855:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 4565 previous similar messages [ 3267.777878] Lustre: 133855:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0 [ 3267.781726] Lustre: 133855:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 4565 previous similar messages [ 3267.785224] Lustre: 133855:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 3267.788376] Lustre: 133855:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 4565 previous similar messages [ 3267.791748] Lustre: 133855:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/39/1, punch: 0/0/0, quota 1/3/0 [ 3267.795739] Lustre: 133855:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 4565 previous similar messages [ 3267.798821] Lustre: 133855:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 3/50/1, delete: 0/0/0 [ 3267.804128] Lustre: 133855:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 4565 previous similar messages [ 3267.807240] Lustre: 133855:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3267.811993] Lustre: 133855:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 4565 previous similar messages [ 3331.768526] Lustre: 130785:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 525, rollback = 2 [ 3331.779168] Lustre: 130785:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 14081 previous similar messages [ 3331.784166] Lustre: 130785:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0 [ 3331.787446] Lustre: 130785:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 14081 previous similar messages [ 3331.791137] Lustre: 130785:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 3331.794891] Lustre: 130785:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 14081 previous similar messages [ 3331.797967] Lustre: 130785:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/39/0, punch: 0/0/0, quota 1/3/0 [ 3331.800973] Lustre: 130785:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 14081 previous similar messages [ 3331.805914] Lustre: 130785:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 3/50/0, delete: 0/0/0 [ 3331.808137] Lustre: 130785:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 14081 previous similar messages [ 3331.812428] Lustre: 130785:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3331.815892] Lustre: 130785:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 14081 previous similar messages [ 3459.768203] Lustre: 134552:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 525, rollback = 2 [ 3459.771801] Lustre: 134552:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 37346 previous similar messages [ 3459.784155] Lustre: 134552:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/1, destroy: 0/0/0 [ 3459.786877] Lustre: 134552:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 37352 previous similar messages [ 3459.794087] Lustre: 134552:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 3459.797197] Lustre: 134552:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 37355 previous similar messages [ 3459.800076] Lustre: 134552:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/39/0, punch: 0/0/0, quota 1/3/0 [ 3459.801988] Lustre: 134552:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 37355 previous similar messages [ 3459.809481] Lustre: 134552:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 3/50/0, delete: 0/0/0 [ 3459.811842] Lustre: 134552:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 37358 previous similar messages [ 3459.814754] Lustre: 134552:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3459.817543] Lustre: 134552:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 37358 previous similar messages [ 3715.772379] Lustre: 130784:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 525, rollback = 2 [ 3715.776803] Lustre: 130784:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 104510 previous similar messages [ 3715.789352] Lustre: 130784:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/1, destroy: 0/0/0 [ 3715.792405] Lustre: 130784:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 104507 previous similar messages [ 3715.795559] Lustre: 130784:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 3715.799456] Lustre: 130784:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 104504 previous similar messages [ 3715.803546] Lustre: 130784:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/39/0, punch: 0/0/0, quota 1/3/0 [ 3715.807073] Lustre: 130784:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 104504 previous similar messages [ 3715.810784] Lustre: 130784:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 3/50/0, delete: 0/0/0 [ 3715.814622] Lustre: 130784:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 104501 previous similar messages [ 3715.818107] Lustre: 130784:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3715.821450] Lustre: 130784:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 104501 previous similar messages [ 4274.042969] Lustre: lustre-MDD0001: changelog off [ 4275.940126] Lustre: 134552:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4276.764163] Lustre: lustre-MDD0000: changelog off [ 4285.925157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4285.931248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4285.933915] LustreError: Skipped 1 previous similar message [ 4285.935713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4285.942946] Lustre: Skipped 4 previous similar messages [ 4285.953114] Lustre: Skipped 11 previous similar messages [ 4296.167255] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4296.239125] LustreError: 140383:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4296.250327] LustreError: 140383:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4296.542907] Lustre: server umount lustre-OST0000 complete [ 4301.285128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4306.312356] Lustre: server umount lustre-MDT0000 complete [ 4306.401827] LustreError: 131740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4306.413253] LustreError: 131740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4309.133540] LustreError: 131559:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764795218 with bad export cookie 13374808828830865885 [ 4309.139667] LustreError: 131559:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4309.140545] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4309.454168] Lustre: server umount lustre-MDT0001 complete [ 4317.367387] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 4319.414777] Key type lgssc unregistered [ 4319.654151] LNet: 141319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4319.664420] LNetError: 141319:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4319.682849] LNet: Removed LNI 192.168.203.120@tcp [ 4320.306580] Key type .llcrypt unregistered [ 4320.310094] Key type ._llcrypt unregistered [ 4336.716269] Key type ._llcrypt registered [ 4336.717779] Key type .llcrypt registered [ 4336.788672] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 4347.745497] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4348.478483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4348.491040] alg: No test for adler32 (adler32-zlib) [ 4349.449106] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 4349.598163] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 4351.241627] Key type lgssc registered [ 4351.888097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4358.718633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4364.899459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4371.194387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4377.672185] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4383.689048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4383.740633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4383.754306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4384.991045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4385.027428] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4385.127794] Lustre: lustre-MDT0000: new disk, initializing [ 4385.220731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4385.249832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4388.782769] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4396.812706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4396.863320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4396.922057] Lustre: 144482:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4396.974536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4396.979991] Lustre: Skipped 1 previous similar message [ 4397.056379] Lustre: lustre-MDT0001: new disk, initializing [ 4397.151521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4397.195787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4397.204468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4400.251728] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4407.009886] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4410.147445] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4416.441873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4416.530120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4416.712678] Lustre: lustre-OST0000: new disk, initializing [ 4416.719601] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4416.802397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4421.481222] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4422.667753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4422.679602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4422.775565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4427.423382] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4431.390713] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_module obdecho/obdecho [ 4432.930121] LustreError: 147102:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4432.936697] LustreError: 147102:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 4434.288560] LustreError: 147199:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 4443.105050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4443.114502] Lustre: Skipped 1 previous similar message [ 4443.115115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4443.120275] Lustre: Skipped 1 previous similar message [ 4448.227315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4448.234662] Lustre: Skipped 3 previous similar messages [ 4448.641334] LustreError: 147539:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4448.770937] Lustre: server umount lustre-MDT0000 complete [ 4451.847612] LustreError: 144475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764795361 with bad export cookie 11966858471123677880 [ 4451.848498] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4451.876928] LustreError: 144475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4452.042956] LustreError: 147740:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4452.046470] LustreError: 147740:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4452.280612] Lustre: server umount lustre-MDT0001 complete [ 4465.993570] LustreError: 147942:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4465.996219] LustreError: 147942:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4466.213562] Lustre: server umount lustre-OST0000 complete [ 4484.340307] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 4491.593919] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4499.103557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4504.127301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4508.447073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4514.237828] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4519.482277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4519.551079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4519.790795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4519.832550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4519.879669] Lustre: lustre-MDT0000: new disk, initializing [ 4519.973262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4520.004908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4522.738960] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4531.429989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4531.497866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4531.565922] Lustre: 150889:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4531.613686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4531.617598] Lustre: Skipped 1 previous similar message [ 4531.705262] Lustre: lustre-MDT0001: new disk, initializing [ 4531.789501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4531.823606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4531.842682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4535.823573] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4543.352933] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4547.341427] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4553.400396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4553.446329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4553.625326] Lustre: lustre-OST0000: new disk, initializing [ 4553.628451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4553.715672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4558.110376] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4563.503127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4563.514214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4563.604913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4563.837644] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4567.284730] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4567.429813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4570.535527] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4570.679688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4573.670145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4573.676161] Lustre: Skipped 1 previous similar message [ 4573.678863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4573.681706] Lustre: Skipped 1 previous similar message [ 4578.302755] LustreError: 153829:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4578.306263] LustreError: 153829:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4578.500168] Lustre: server umount lustre-OST0000 complete [ 4582.881371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4582.890435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.904160] Lustre: Skipped 1 previous similar message [ 4582.911220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4587.761429] LustreError: 154031:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4587.766158] LustreError: 154031:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4587.873790] Lustre: server umount lustre-MDT0000 complete [ 4590.820224] LustreError: 150881:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764795500 with bad export cookie 11966858471123678839 [ 4590.826809] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4590.837202] LustreError: 150881:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4591.215514] Lustre: server umount lustre-MDT0001 complete [ 4594.722809] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 15:58:22 (1764795502) [ 4596.082712] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4597.518856] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 15:58:25 (1764795505) [ 4604.004429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4604.289126] LustreError: 154903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4604.299580] LustreError: 154903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4604.343963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4607.409570] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4609.507875] LustreError: 154903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4614.625821] LustreError: 154903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4615.095050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4619.200857] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4624.355759] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4627.220710] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4634.029961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4639.458766] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4643.865918] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4655.108395] Lustre: 156867:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 504 < left 1021, rollback = 2 [ 4655.122348] Lustre: 156867:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 4/16/8, destroy: 0/0/0 [ 4655.138413] Lustre: 156867:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 9/1021/0 [ 4655.154464] Lustre: 156867:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/153/0, punch: 0/0/0, quota 1/3/0 [ 4655.158111] Lustre: 156867:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 13/232/4, delete: 0/0/0 [ 4655.187408] Lustre: 156867:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 5/5/0, ref_del: 0/0/0 [ 4661.731305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4661.735449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4661.747462] Lustre: Skipped 1 previous similar message [ 4661.754266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4661.760226] Lustre: Skipped 2 previous similar messages [ 4665.936711] LustreError: 157173:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4665.939494] LustreError: 157173:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4666.038194] Lustre: server umount lustre-MDT0000 complete [ 4669.497050] LustreError: 154884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764795579 with bad export cookie 11966858471123679714 [ 4669.500175] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4669.506571] LustreError: 154884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4670.032830] LustreError: 154898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4670.066671] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.20@tcp (stopping) [ 4670.075481] Lustre: Skipped 4 previous similar messages [ 4670.436774] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4670.443355] Lustre: Skipped 2 previous similar messages [ 4674.464140] LustreError: 154899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4674.480281] LustreError: 154899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4675.910292] Lustre: server umount lustre-MDT0001 complete [ 4680.931715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4690.051713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4690.184843] LustreError: 158434:0:(llog.c:1616:llog_backup()) MGC192.168.203.120@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4690.190763] Lustre: 158434:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.120@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4700.128313] LustreError: 158442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4700.144474] LustreError: 158442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4719.585767] LustreError: 158442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4719.594698] LustreError: 158442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 4751.327367] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4751.329982] Lustre: 158434:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.120@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4751.945673] LustreError: 158442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4751.961569] LustreError: 158442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [ 4779.999339] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4780.018199] Lustre: 154895:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.120@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4791.887674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4791.900260] Lustre: Skipped 2 previous similar messages [ 4794.914677] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4796.322543] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4800.484191] LustreError: 154895:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 4804.390483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4814.306505] LustreError: 142692:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ef222e24 x1850521953620608/t0(0) o250->MGC192.168.203.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4814.334980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 4819.040799] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4819.957912] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4824.330485] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4825.826711] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4829.095726] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4830.538286] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4850.660075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4850.662707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4850.663250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4850.682907] Lustre: Skipped 2 previous similar messages [ 4850.685553] Lustre: Skipped 4 previous similar messages [ 4853.913779] LustreError: 159954:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4853.927932] LustreError: 159954:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 4854.092765] Lustre: server umount lustre-MDT0000 complete [ 4855.786183] LustreError: 158441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4855.817229] LustreError: 158441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 48 previous similar messages [ 4857.746640] LustreError: 154883:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764795767 with bad export cookie 11966858471123686231 [ 4857.752183] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4857.762920] LustreError: 154883:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4858.053607] Lustre: server umount lustre-MDT0001 complete [ 4871.979733] Lustre: server umount lustre-OST0000 complete [ 4888.107474] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 4894.785582] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4901.747468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4907.244773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4913.512329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4920.366086] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4927.058560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4927.109727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4927.304650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4927.334694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4927.390353] Lustre: lustre-MDT0000: new disk, initializing [ 4927.462512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4927.465864] Lustre: Skipped 1 previous similar message [ 4927.482280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4930.502819] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4938.667037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4938.727902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4938.795641] Lustre: 163304:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4939.017715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4942.185989] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4948.711478] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4951.865464] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4960.265630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4960.362340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4960.586632] Lustre: lustre-OST0000: new disk, initializing [ 4960.589145] Lustre: Skipped 1 previous similar message [ 4960.594685] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4960.597815] Lustre: Skipped 2 previous similar messages [ 4966.014083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4966.025518] Lustre: Skipped 1 previous similar message [ 4966.027926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4966.194155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4968.392058] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4976.801744] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4981.038382] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4981.331262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4984.820132] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4985.008548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4986.849867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4986.853914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4986.856428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4986.861322] Lustre: Skipped 1 previous similar message [ 4986.864161] Lustre: Skipped 1 previous similar message [ 4992.730149] LustreError: 166244:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4992.732848] LustreError: 166244:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4992.928935] Lustre: server umount lustre-OST0000 complete [ 5002.459094] Lustre: server umount lustre-MDT0000 complete [ 5005.634382] LustreError: 165049:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764795915 with bad export cookie 11966858471123703024 [ 5005.640125] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5005.689315] LustreError: 165049:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5006.183840] Lustre: server umount lustre-MDT0001 complete [ 5013.944790] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 16:05:21 (1764795921) [ 5019.423517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5019.824127] LustreError: 167318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5023.060412] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5030.165466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5033.927626] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5038.272631] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5041.526597] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5047.083505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5052.126131] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5056.518939] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5074.735523] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 16:06:22 (1764795982) [ 5080.038891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5080.039642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5080.044419] Lustre: Skipped 2 previous similar messages [ 5080.045549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5080.045555] Lustre: Skipped 7 previous similar messages [ 5080.060669] LustreError: Skipped 1 previous similar message [ 5083.138766] LustreError: 169777:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5083.141996] LustreError: 169777:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5083.303919] Lustre: server umount lustre-OST0000 complete [ 5092.247232] Lustre: server umount lustre-MDT0000 complete [ 5095.371371] LustreError: 167297:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764796005 with bad export cookie 11966858471123703906 [ 5095.377979] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5095.416445] LustreError: 167297:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5095.892646] Lustre: server umount lustre-MDT0001 complete [ 5103.566399] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5105.352978] Key type lgssc unregistered [ 5105.560990] LNet: 170714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5105.568631] LNetError: 170714:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5105.586945] LNet: Removed LNI 192.168.203.120@tcp [ 5106.450285] Key type .llcrypt unregistered [ 5106.454302] Key type ._llcrypt unregistered [ 5120.169791] Key type ._llcrypt registered [ 5120.171570] Key type .llcrypt registered [ 5120.241751] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 5120.881848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5120.959973] alg: No test for adler32 (adler32-zlib) [ 5121.929339] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 5122.081906] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 5123.703221] Key type lgssc registered [ 5124.680208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5130.686417] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5130.698599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5146.207293] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5151.329324] LustreError: 171916:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.120@tcp: failed processing log, type 4: rc = -110 [ 5156.447401] LustreError: 171916:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5156.456266] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5156.463708] LustreError: MGC192.168.203.120@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5156.468682] LustreError: 171916:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5156.475542] LustreError: 171916:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 5156.479660] LustreError: 171916:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 5156.483846] LustreError: 171916:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 5156.587579] Lustre: server umount lustre-OST0000 complete [ 5156.589511] LustreError: 171916:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5163.246388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.575110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5167.719310] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5174.574150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5174.753986] LustreError: 172417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5174.770080] LustreError: 172417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5174.986808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5178.418980] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5183.710991] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5186.745069] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5192.542930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5192.686504] LustreError: 174008:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5192.700308] Lustre: 174008:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.203.120@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5192.843038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5198.066130] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5202.768818] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5204.443029] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.451602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5204.464308] Lustre: Skipped 2 previous similar messages [ 5204.466832] Lustre: Skipped 1 previous similar message [ 5209.573954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5209.577072] Lustre: Skipped 1 previous similar message [ 5210.582641] LustreError: 174660:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5210.585854] LustreError: 174660:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5210.674396] Lustre: server umount lustre-MDT0000 complete [ 5213.281519] LustreError: 172399:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764796123 with bad export cookie 5810624964715196999 [ 5213.285910] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5213.291680] LustreError: 172399:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5213.385335] LustreError: 174862:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5213.387924] LustreError: 174862:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5213.525962] Lustre: server umount lustre-MDT0001 complete [ 5227.077083] LustreError: 175065:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5227.091353] LustreError: 175065:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5227.341365] Lustre: server umount lustre-OST0000 complete [ 5235.439422] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 5243.403217] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 5250.803236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5256.209289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5260.475620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5265.930335] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5272.535188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5272.607893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5272.880188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5272.937773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5273.080606] Lustre: lustre-MDT0000: new disk, initializing [ 5273.197113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5273.219467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5277.837845] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5288.488921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5288.691727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5288.855120] Lustre: 177407:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5288.969603] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5288.973385] Lustre: Skipped 1 previous similar message [ 5289.123751] Lustre: lustre-MDT0001: new disk, initializing [ 5289.258937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5289.313395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5289.329112] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5294.332189] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5302.596470] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5306.994746] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5314.673102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5314.711983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5315.007687] Lustre: lustre-OST0000: new disk, initializing [ 5315.013203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5315.095825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5321.696274] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5322.804093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5322.821365] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5322.955735] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5329.054560] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5333.795424] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5333.992742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5338.114620] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5338.309864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5343.712990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5343.713290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5343.722016] LustreError: Skipped 1 previous similar message [ 5343.722491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5343.743970] Lustre: Skipped 1 previous similar message [ 5346.045132] LustreError: 180342:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5346.047914] LustreError: 180342:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5346.280942] Lustre: server umount lustre-OST0000 complete [ 5350.881955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5350.904399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5350.920650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5350.925609] Lustre: Skipped 1 previous similar message [ 5353.956527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5355.794684] LustreError: 180544:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5355.797797] LustreError: 180544:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5355.855128] Lustre: server umount lustre-MDT0000 complete [ 5359.072309] LustreError: 180344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5359.081748] LustreError: 180344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5359.572620] LustreError: 177398:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764796269 with bad export cookie 5810624964715197713 [ 5359.582454] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5359.611927] LustreError: 177398:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5360.167839] Lustre: server umount lustre-MDT0001 complete [ 5369.556133] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 16:11:17 (1764796277) [ 5385.833928] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5388.356764] Key type lgssc unregistered [ 5388.642216] LNet: 181933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5388.659525] LNetError: 181933:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5389.681967] LNet: Removed LNI 192.168.203.120@tcp [ 5390.600988] Key type .llcrypt unregistered [ 5390.602720] Key type ._llcrypt unregistered [ 5406.046074] Key type ._llcrypt registered [ 5406.047527] Key type .llcrypt registered [ 5406.141761] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 5407.224718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5407.243843] alg: No test for adler32 (adler32-zlib) [ 5408.329238] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 5408.520781] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 5410.204341] Key type lgssc registered [ 5411.179103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5417.155970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5417.172037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5418.526875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5421.427522] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5428.002782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5428.212509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5431.243045] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5435.038188] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5438.003279] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5443.830857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5443.962383] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5443.966223] LustreError: 184769:0:(llog.c:1657:llog_backup()) MGC192.168.203.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5443.973585] Lustre: 184769:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.203.120@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5444.072928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5449.061368] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5454.792057] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5457.379257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5457.383430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5457.402467] Lustre: Skipped 2 previous similar messages [ 5462.502115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5462.506851] Lustre: Skipped 3 previous similar messages [ 5463.253498] LustreError: 185421:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5463.393531] Lustre: server umount lustre-MDT0000 complete [ 5466.886055] LustreError: 183113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764796376 with bad export cookie 3365985755226612218 [ 5466.893510] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5466.925036] LustreError: 183113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5467.064438] LustreError: 185624:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5467.066623] LustreError: 185624:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5467.263594] Lustre: server umount lustre-MDT0001 complete [ 5480.526359] LustreError: 185824:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5480.528993] LustreError: 185824:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5480.743522] Lustre: server umount lustre-OST0000 complete [ 5488.732179] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 5497.261259] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 5505.650254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5510.927374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5515.764316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5521.530274] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5527.313597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5527.394128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5527.683083] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5527.719366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5527.841965] Lustre: lustre-MDT0000: new disk, initializing [ 5527.918449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5527.930160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5532.187096] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5542.481189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5542.553227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5542.650128] Lustre: 188162:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5542.697227] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5542.699218] Lustre: Skipped 1 previous similar message [ 5542.703813] LustreError: 188174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5542.707812] LustreError: 188929:0:(llog.c:1657:llog_backup()) MGC192.168.203.120@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5542.707821] LustreError: 188929:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 5542.707857] Lustre: 188929:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.120@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5542.727556] LustreError: 188174:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5542.745973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5542.747990] Lustre: Skipped 1 previous similar message [ 5542.807858] Lustre: lustre-MDT0001: new disk, initializing [ 5542.995404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5543.042684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5543.048393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5547.477477] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5555.267734] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5559.076708] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5565.691384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5565.776458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5566.013599] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5566.015435] Lustre: Skipped 1 previous similar message [ 5566.017354] LustreError: 189882:0:(llog.c:1657:llog_backup()) MGC192.168.203.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5566.021164] LustreError: 189882:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 5566.024567] Lustre: 189882:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.120@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5566.029114] Lustre: 189882:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5566.043145] Lustre: lustre-OST0000: new disk, initializing [ 5566.060492] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5566.180085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5571.493328] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5574.225580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5574.232907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5574.438862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5579.235530] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5583.141599] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5583.315675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5587.085385] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5587.353693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5589.987506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5589.988041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5589.989106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5590.002648] Lustre: Skipped 1 previous similar message [ 5590.025306] LustreError: Skipped 1 previous similar message [ 5595.106821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5595.109521] Lustre: Skipped 2 previous similar messages [ 5595.192534] LustreError: 191101:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5595.195031] LustreError: 191101:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5595.343331] Lustre: server umount lustre-OST0000 complete [ 5599.201246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5599.205946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5599.218395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5604.389216] LustreError: 191303:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5604.393334] LustreError: 191303:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5604.490635] Lustre: server umount lustre-MDT0000 complete [ 5605.346163] LustreError: 191102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5605.356509] LustreError: 191102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5607.472885] LustreError: 188153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764796517 with bad export cookie 3365985755226612939 [ 5607.478679] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5607.485570] LustreError: 188153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5607.781115] Lustre: server umount lustre-MDT0001 complete [ 5615.046373] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 16:15:22 (1764796522) [ 5629.733803] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 5639.886601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5640.191631] LustreError: 192875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5640.235341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5644.034293] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5645.282596] LustreError: 192875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5650.405605] LustreError: 192875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5651.238496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5655.321851] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5658.028225] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5663.575932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5668.909951] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5677.789313] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5677.872827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5678.003127] Lustre: lustre-OST0001: new disk, initializing [ 5678.005957] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5678.118525] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5678.121828] Lustre: Skipped 2 previous similar messages [ 5679.380698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5679.400895] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5679.531258] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5683.599618] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5694.701347] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5703.051546] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5708.445660] Lustre: 192870:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 5708.461446] Lustre: 192870:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5708.475294] Lustre: 192870:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 5708.482429] Lustre: 192870:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/153/0, punch: 0/0/0, quota 1/3/0 [ 5708.487500] Lustre: 192870:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5708.494267] Lustre: 192870:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5710.367838] Lustre: DEBUG MARKER: ADD OST3 [ 5715.463849] loop: module loaded [ 5715.511787] loop0: detected capacity change from 0 to 204800000 [ 5715.729240] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5715.744885] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5715.760500] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5715.886502] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5721.870611] loop0: detected capacity change from 0 to 204800000 [ 5722.520751] Lustre: DEBUG MARKER: STOP OST3 [ 5726.927633] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5727.051346] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5727.284909] Lustre: lustre-OST0003: new disk, initializing [ 5727.299569] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5728.691335] Lustre: *** cfs_fail_loc=2109, val=2*** [ 5728.869361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5728.873605] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5728.948229] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 5730.720066] LustreError: 197278:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 5731.225471] Lustre: Failing over lustre-OST0003 [ 5731.297101] LustreError: 197484:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 5731.304653] LustreError: 197484:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5731.415534] Lustre: server umount lustre-OST0003 complete [ 5733.494067] LustreError: 194330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5733.859673] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5733.872071] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5733.913992] Lustre: Skipped 2 previous similar messages [ 5734.344937] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5744.463446] Lustre: DEBUG MARKER: START OST3 again [ 5750.532951] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5750.931261] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5750.940537] Lustre: Skipped 1 previous similar message [ 5750.958374] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5752.061722] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5752.373679] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 5752.375472] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5755.409304] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5777.467217] Lustre: Failing over lustre-OST0003 [ 5777.567599] LustreError: 199485:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 5777.591773] LustreError: 199485:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5777.649604] Lustre: server umount lustre-OST0003 complete [ 5778.400324] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5778.402949] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5778.411074] LustreError: 195228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5778.445925] LustreError: 195228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 5778.457353] Lustre: Skipped 1 previous similar message [ 5797.347650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5797.351279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5797.355370] Lustre: Skipped 2 previous similar messages [ 5797.384079] Lustre: Skipped 5 previous similar messages [ 5801.227311] Lustre: server umount lustre-MDT0000 complete [ 5804.676046] LustreError: 192854:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764796714 with bad export cookie 3365985755226613821 [ 5804.681392] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5810.868103] LustreError: 199957:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 5810.871261] LustreError: 199957:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5811.125191] Lustre: server umount lustre-MDT0001 complete [ 5814.326950] Lustre: server umount lustre-OST0000 complete [ 5817.760553] Lustre: server umount lustre-OST0001 complete [ 5823.242591] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 5831.026435] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 5840.559546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5847.132678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5852.469782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5858.552174] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5866.269871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5866.341886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5866.525849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5866.548408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5866.602466] Lustre: lustre-MDT0000: new disk, initializing [ 5866.681956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5871.510479] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5883.224645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5883.307823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5883.435295] Lustre: 202547:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5883.520698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5883.526212] Lustre: Skipped 1 previous similar message [ 5883.566929] Lustre: lustre-MDT0001: new disk, initializing [ 5883.750599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5883.770901] Lustre: Skipped 1 previous similar message [ 5883.822876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5883.850315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5889.107096] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5898.648256] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5903.320520] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5910.613516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5910.684290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5912.185139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5912.348807] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5916.372028] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5923.425373] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5926.907233] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5927.079306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5930.730413] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5930.931619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5933.027090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5933.029850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5933.045171] LustreError: Skipped 2 previous similar messages [ 5933.047811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5933.047819] Lustre: Skipped 4 previous similar messages [ 5933.060987] Lustre: Skipped 4 previous similar messages [ 5938.628105] LustreError: 205488:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5938.631201] LustreError: 205488:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 5938.845263] Lustre: server umount lustre-OST0000 complete [ 5948.507806] Lustre: server umount lustre-MDT0000 complete [ 5951.553674] LustreError: 202538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764796861 with bad export cookie 3365985755226616110 [ 5951.564842] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5951.566417] LustreError: 202538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5951.868457] Lustre: server umount lustre-MDT0001 complete [ 5954.807241] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 16:21:02 (1764796862) [ 5967.715731] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 5975.224300] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 5982.790817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5989.059898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5994.362705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6000.617950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6007.145628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6007.223639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6007.407809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6007.452569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6007.456474] Lustre: Skipped 1 previous similar message [ 6007.527860] Lustre: lustre-MDT0000: new disk, initializing [ 6007.530223] Lustre: Skipped 1 previous similar message [ 6007.612471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6007.624317] Lustre: Skipped 1 previous similar message [ 6010.992379] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6020.887545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6020.961556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6021.066573] Lustre: 208737:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6021.402540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6024.918604] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6031.078110] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6034.589920] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6040.555153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6040.618141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6042.931284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6045.819137] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6053.430571] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6057.316867] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6057.487662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6060.863138] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6061.088524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6063.589749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6063.596965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6063.605873] Lustre: Skipped 2 previous similar messages [ 6063.613387] LustreError: Skipped 1 previous similar message [ 6063.615390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6063.617939] Lustre: Skipped 7 previous similar messages [ 6068.807085] LustreError: 211675:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6068.814303] LustreError: 211675:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6068.984910] Lustre: server umount lustre-OST0000 complete [ 6078.369571] Lustre: server umount lustre-MDT0000 complete [ 6078.946859] LustreError: 208749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6078.955342] LustreError: 208749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 6081.487610] LustreError: 208728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764796991 with bad export cookie 3365985755226616999 [ 6081.488822] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6081.500994] LustreError: 208728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6081.932588] Lustre: server umount lustre-MDT0001 complete [ 6089.644854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6093.349290] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6101.289390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6105.311373] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6110.909882] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6114.475372] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6120.911761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6126.506489] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6131.390445] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6147.042420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6147.051529] LustreError: Skipped 1 previous similar message [ 6147.058362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.069542] Lustre: Skipped 3 previous similar messages [ 6147.082563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6147.085526] Lustre: Skipped 6 previous similar messages [ 6152.440955] Lustre: server umount lustre-OST0000 complete [ 6161.649426] Lustre: server umount lustre-MDT0000 complete [ 6164.817974] LustreError: 212538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764797074 with bad export cookie 3365985755226617881 [ 6164.831803] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6165.173543] Lustre: server umount lustre-MDT0001 complete [ 6267.267516] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 6269.133873] Key type lgssc unregistered [ 6269.423222] LNet: 215886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6269.434982] LNetError: 215886:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6269.449749] LNet: Removed LNI 192.168.203.120@tcp [ 6270.081166] Key type .llcrypt unregistered [ 6270.083961] Key type ._llcrypt unregistered [ 6288.915159] Key type ._llcrypt registered [ 6288.917861] Key type .llcrypt registered [ 6289.016766] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 6301.284490] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 6302.121352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6302.146671] alg: No test for adler32 (adler32-zlib) [ 6303.179431] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 6303.427364] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 6305.113755] Key type lgssc registered [ 6306.378220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6314.073941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6319.352705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6324.753621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6330.155698] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6336.347877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6336.400823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6336.418523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6337.621397] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6337.644437] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6337.694864] Lustre: lustre-MDT0000: new disk, initializing [ 6337.775985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6337.797907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6340.937749] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6350.008387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6350.080822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6350.144485] Lustre: 219194:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6350.222419] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6350.228217] Lustre: Skipped 1 previous similar message [ 6350.297862] Lustre: lustre-MDT0001: new disk, initializing [ 6350.438791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6350.526147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6350.532089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6353.746527] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6359.783979] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6362.887925] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6368.657387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6368.790780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6369.214854] Lustre: lustre-OST0000: new disk, initializing [ 6369.217728] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6369.301975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6374.390973] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6377.518543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6377.522801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6377.592213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6380.959677] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6384.933379] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6385.076447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6388.054609] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6388.300821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6392.817107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6392.823078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6392.824535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6392.852133] Lustre: Skipped 1 previous similar message [ 6396.428968] LustreError: 222131:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6396.661977] Lustre: server umount lustre-OST0000 complete [ 6401.508652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6401.514457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6401.522371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6401.528476] Lustre: Skipped 1 previous similar message [ 6403.040442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6403.040959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6408.161394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6408.171735] Lustre: Skipped 2 previous similar messages [ 6412.258615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6412.271168] Lustre: Skipped 1 previous similar message [ 6414.303392] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6414.405851] LustreError: 222332:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6414.407859] LustreError: 222332:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6414.536905] Lustre: server umount lustre-MDT0000 complete [ 6418.084837] LustreError: 219184:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764797327 with bad export cookie 2372889945733310097 [ 6418.089290] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6418.093701] LustreError: 219184:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6418.243552] LustreError: 222535:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6418.256119] LustreError: 222535:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6418.748858] Lustre: server umount lustre-MDT0001 complete [ 6435.931521] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 6445.196806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6445.477102] LustreError: 223714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6445.494780] LustreError: 223714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6445.552973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6448.924084] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6450.661562] LustreError: 223714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6455.386358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6455.638330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6458.494864] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6460.593442] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6465.230817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6465.485662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6469.850437] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6476.349683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6476.400227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6476.502497] Lustre: lustre-OST0001: new disk, initializing [ 6476.506280] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6480.468105] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6485.575893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6485.584521] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6485.678873] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6490.864757] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6500.575956] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6515.072587] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 16:30:23 (1764797423) [ 6517.217951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6517.222158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6517.234682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6521.826732] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6521.833211] Lustre: Skipped 2 previous similar messages [ 6522.979439] LustreError: 227092:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6522.982850] LustreError: 227092:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6523.047206] Lustre: server umount lustre-MDT0000 complete [ 6525.827852] LustreError: 225987:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764797435 with bad export cookie 2372889945733310979 [ 6525.829240] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6525.845887] LustreError: 225987:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6526.145798] Lustre: server umount lustre-MDT0001 complete [ 6539.141180] LustreError: 227494:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6539.147805] LustreError: 227494:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 6539.218036] Lustre: server umount lustre-OST0000 complete [ 6551.808791] LustreError: 227696:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6551.811821] LustreError: 227696:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6552.102353] Lustre: server umount lustre-OST0001 complete [ 6557.593410] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 6566.989769] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 6574.985679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6580.369315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6585.388691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6590.610201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6596.040408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6596.081964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6596.236300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6596.262358] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6596.316603] Lustre: lustre-MDT0000: new disk, initializing [ 6596.407167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6596.411639] Lustre: Skipped 1 previous similar message [ 6596.443945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6599.251806] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6607.494606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6607.637855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6607.800472] Lustre: 229885:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6607.901421] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6607.918983] Lustre: Skipped 1 previous similar message [ 6608.030486] Lustre: lustre-MDT0001: new disk, initializing [ 6608.205236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6608.215845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6611.782474] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6619.301551] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6622.795797] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6630.030799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6630.104705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6630.326931] Lustre: lustre-OST0000: new disk, initializing [ 6630.329653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6630.402796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6630.407022] Lustre: Skipped 1 previous similar message [ 6631.819514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6631.826857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6631.931707] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6635.891158] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6642.544316] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6645.916801] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6646.053368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6649.102604] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6649.283601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6652.391917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6652.393696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6652.403412] LustreError: Skipped 1 previous similar message [ 6652.403429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6652.415484] Lustre: Skipped 4 previous similar messages [ 6665.184256] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6665.230791] LustreError: 232826:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6665.233122] LustreError: 232826:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6665.417266] Lustre: server umount lustre-OST0000 complete [ 6669.799599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6669.802956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.842972] Lustre: Skipped 1 previous similar message [ 6683.103127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6683.415842] Lustre: server umount lustre-MDT0000 complete [ 6687.082411] LustreError: 229878:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764797596 with bad export cookie 2372889945733313121 [ 6687.090612] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6687.105502] LustreError: 229878:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6687.519180] Lustre: server umount lustre-MDT0001 complete [ 6696.563644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6696.938722] LustreError: 233708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6697.011326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6701.236844] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6702.056113] LustreError: 233708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6707.180993] LustreError: 233708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.429619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6713.342939] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6717.344883] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6720.223876] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6726.189783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6731.664570] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6736.809846] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6883.296161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6883.299140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6883.327470] Lustre: Skipped 1 previous similar message [ 6883.331849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6883.334415] Lustre: Skipped 11 previous similar messages [ 6888.327288] LustreError: 235995:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6888.329878] LustreError: 235995:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6888.663440] Lustre: server umount lustre-OST0000 complete [ 6898.937878] Lustre: server umount lustre-MDT0000 complete [ 6901.732753] LustreError: 233708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6901.750739] LustreError: 233708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6903.100042] LustreError: 233689:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764797812 with bad export cookie 2372889945733314003 [ 6903.116204] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6903.497748] Lustre: server umount lustre-MDT0001 complete [ 7005.910443] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 7008.989962] Key type lgssc unregistered [ 7009.265770] LNet: 236965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7009.273990] LNetError: 236965:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7009.293592] LNet: Removed LNI 192.168.203.120@tcp [ 7010.087157] Key type .llcrypt unregistered [ 7010.089786] Key type ._llcrypt unregistered [ 7037.224752] Key type ._llcrypt registered [ 7037.226378] Key type .llcrypt registered [ 7037.351365] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 7055.396446] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7056.720562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7056.740182] alg: No test for adler32 (adler32-zlib) [ 7057.709709] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 7057.869662] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 7059.599220] Key type lgssc registered [ 7060.856640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7069.878944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7077.269965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7084.763917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7092.894231] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7110.933539] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7120.094847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7120.172548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7120.197544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7121.452962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7121.484195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7121.540978] Lustre: lustre-MDT0000: new disk, initializing [ 7121.654135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7121.679513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7125.286602] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7135.592673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7135.642507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7135.731464] Lustre: 240995:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7135.812504] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7135.818564] Lustre: Skipped 1 previous similar message [ 7135.870742] Lustre: lustre-MDT0001: new disk, initializing [ 7135.936211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7135.968538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7135.980751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7139.955364] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7144.579694] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7151.028527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7151.077846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7151.276704] Lustre: lustre-OST0000: new disk, initializing [ 7151.289216] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7151.377163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7154.230761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7154.247748] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7154.403219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7157.186651] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7166.679834] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7166.767161] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7166.905781] Lustre: lustre-OST0001: new disk, initializing [ 7166.909543] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7166.993481] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7171.594795] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7174.209588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7174.225464] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7174.330271] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7181.693923] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7185.682143] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7200.227973] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7200.233557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7200.260687] Lustre: Skipped 2 previous similar messages [ 7200.262814] Lustre: Skipped 3 previous similar messages [ 7205.351912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7205.362504] Lustre: Skipped 4 previous similar messages [ 7205.514982] LustreError: 244522:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7205.605513] Lustre: server umount lustre-MDT0000 complete [ 7208.928245] LustreError: 240986:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764798118 with bad export cookie 10063890276759871716 [ 7208.931159] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7208.950774] LustreError: 240986:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7209.125586] LustreError: 244724:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7209.128705] LustreError: 244724:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7209.318898] Lustre: server umount lustre-MDT0001 complete [ 7222.214318] LustreError: 244924:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7222.216865] LustreError: 244924:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7222.260910] Lustre: server umount lustre-OST0000 complete [ 7234.300862] LustreError: 245127:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7234.304738] LustreError: 245127:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7234.455298] Lustre: server umount lustre-OST0001 complete [ 7245.582200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7253.287846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7254.133904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7261.588434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7271.025119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7272.061178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7291.267935] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7300.525733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7300.546115] Lustre: lustre-MDT0000: reset Object Index mappings [ 7301.018836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7304.298790] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7306.218210] LustreError: 248442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7306.245434] LustreError: 248442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7310.305499] LustreError: 248442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7312.377735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7312.393309] Lustre: lustre-MDT0001: reset Object Index mappings [ 7312.820571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7316.223162] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7318.500777] Lustre: 249551:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7324.265812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7325.604859] LustreError: 249908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7325.619150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7329.047613] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7330.789491] LustreError: 249908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7335.921602] LustreError: 249908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7336.804298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7337.174929] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7337.188150] Lustre: Skipped 1 previous similar message [ 7338.172109] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7343.264898] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7351.285820] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7354.574535] Lustre: 251398:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7364.083289] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7364.092880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7366.312268] LustreError: 251504:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7366.327832] LustreError: 251504:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7366.506763] Lustre: server umount lustre-MDT0000 complete [ 7369.192512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7369.197828] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7369.213603] LustreError: 251418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7369.227617] Lustre: Skipped 2 previous similar messages [ 7370.097945] LustreError: 248421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764798279 with bad export cookie 10063890276759878506 [ 7370.109515] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7370.120179] LustreError: 248421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7370.731366] Lustre: server umount lustre-MDT0001 complete [ 7385.171725] LustreError: 251906:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 7385.180441] LustreError: 251906:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 7385.264750] Lustre: server umount lustre-OST0000 complete [ 7398.491760] Lustre: server umount lustre-OST0001 complete [ 7414.582475] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 16:45:22 (1764798322) [ 7431.915964] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 7441.664354] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7452.528684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7459.864780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7466.928575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7474.610847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7482.776232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7482.843410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7483.077980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7483.105943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7483.178531] Lustre: lustre-MDT0000: new disk, initializing [ 7483.304334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7483.325238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7486.716891] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7496.276295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7496.318416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7496.398609] Lustre: 255637:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7496.468860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7496.472987] Lustre: Skipped 1 previous similar message [ 7496.526888] Lustre: lustre-MDT0001: new disk, initializing [ 7496.662174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7496.685834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7500.119485] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7507.879448] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7511.557947] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7517.520609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7517.613636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7517.851766] Lustre: lustre-OST0000: new disk, initializing [ 7517.858711] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7518.914127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7518.931410] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7519.043132] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7522.324985] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7528.445784] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7531.644494] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7531.790534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7534.590231] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7534.767910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7539.685553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7539.687167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7539.688444] LustreError: Skipped 1 previous similar message [ 7539.701214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7542.591268] LustreError: 258573:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7542.605443] LustreError: 258573:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 7542.743946] Lustre: server umount lustre-OST0000 complete [ 7547.875392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7547.877826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7547.901913] Lustre: Skipped 1 previous similar message [ 7547.905470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7547.908135] Lustre: Skipped 1 previous similar message [ 7560.161628] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7560.162756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7560.186150] Lustre: Skipped 5 previous similar messages [ 7560.471164] Lustre: server umount lustre-MDT0000 complete [ 7565.280702] LustreError: 255643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7565.312047] LustreError: 255643:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7565.698568] LustreError: 255629:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764798475 with bad export cookie 10063890276759880599 [ 7565.708873] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7565.724458] LustreError: 255629:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7566.795322] Lustre: server umount lustre-MDT0001 complete [ 7589.900572] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7599.624497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7599.909464] LustreError: 260156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7599.966192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7599.969134] Lustre: Skipped 2 previous similar messages [ 7603.582573] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7613.295738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7618.359603] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7621.733961] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7629.786733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7635.611260] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7643.452887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7643.527421] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7643.747531] Lustre: lustre-OST0001: new disk, initializing [ 7643.749762] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7645.699402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7645.717168] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7645.862799] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7651.022498] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7661.682854] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7665.398235] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7675.146982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 7675.174077] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 7680.484456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7680.505211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7680.514498] Lustre: Skipped 1 previous similar message [ 7680.519057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7682.361187] LustreError: 263663:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7682.374560] LustreError: 263663:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7682.507210] Lustre: server umount lustre-MDT0000 complete [ 7685.885470] LustreError: 260137:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764798595 with bad export cookie 10063890276759881481 [ 7685.893907] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7686.345552] Lustre: server umount lustre-MDT0001 complete [ 7699.968343] Lustre: server umount lustre-OST0000 complete [ 7702.943138] Lustre: 238506:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764798596/real 1764798596] req@ffff9b5f8566d880 x1850524793267328/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764798612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7702.983276] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7703.010271] Lustre: Skipped 3 previous similar messages [ 7703.573864] Lustre: server umount lustre-OST0001 complete [ 7719.135802] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7729.147830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7729.448974] LustreError: 265447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7729.471332] LustreError: 265447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7729.535327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7729.539952] Lustre: Skipped 3 previous similar messages [ 7732.998194] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7740.235708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7744.323132] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7747.628780] Lustre: 266556:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7754.571810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7763.309211] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7774.748925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 7775.514406] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7777.856465] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 7783.665232] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7791.773225] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7795.192613] Lustre: 268410:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7805.552288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 7805.631425] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 7816.465280] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 16:52:03 (1764798723) [ 7832.550025] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7832.551251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7832.575169] Lustre: Skipped 3 previous similar messages [ 7832.605072] Lustre: Skipped 7 previous similar messages [ 7836.820441] LustreError: 269116:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7836.830263] LustreError: 269116:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 7837.011126] Lustre: server umount lustre-MDT0000 complete [ 7840.861660] LustreError: 265426:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764798750 with bad export cookie 10063890276759887774 [ 7840.866694] LustreError: 265426:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7840.869375] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7847.360825] Lustre: server umount lustre-MDT0001 complete [ 7857.499520] Lustre: server umount lustre-OST0000 complete [ 7862.736414] Lustre: server umount lustre-OST0001 complete [ 7871.362531] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 7881.069802] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7891.121621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7896.888887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7902.843255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7909.175251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7926.834958] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7938.463861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7938.544835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7938.811646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7938.870463] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7938.993787] Lustre: lustre-MDT0000: new disk, initializing [ 7939.178839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7943.549963] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7953.421434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7953.537581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7953.680555] Lustre: 272610:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7954.063539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7958.538482] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7969.899359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7969.963124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7971.920241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7975.581603] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7986.861479] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7987.028297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7987.491451] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7987.505537] Lustre: Skipped 6 previous similar messages [ 7988.967710] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7988.982778] Lustre: Skipped 1 previous similar message [ 7989.216618] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7994.799877] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8005.422712] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8009.283313] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8009.285925] Lustre: Skipped 1 previous similar message [ 8029.380509] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 16:55:37 (1764798937) [ 8033.515208] LustreError: 276223:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8033.517825] LustreError: 276223:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 8033.686550] Lustre: server umount lustre-MDT0000 complete [ 8035.311167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8035.315534] LustreError: 272618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8035.330094] Lustre: Skipped 3 previous similar messages [ 8035.343841] LustreError: 272618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 8037.499091] LustreError: 272602:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764798947 with bad export cookie 10063890276759895649 [ 8037.505109] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8037.523766] LustreError: 272602:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8037.944163] Lustre: server umount lustre-MDT0001 complete [ 8051.282716] Lustre: server umount lustre-OST0000 complete [ 8065.044788] Lustre: server umount lustre-OST0001 complete [ 8080.337581] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 8087.789891] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 8095.931199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8102.853698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8109.883442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8118.680059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8128.036611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8128.145795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8128.592314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8128.640654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8128.651570] Lustre: Skipped 4 previous similar messages [ 8128.880053] Lustre: lustre-MDT0000: new disk, initializing [ 8128.882034] Lustre: Skipped 3 previous similar messages [ 8129.138877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8129.141837] Lustre: Skipped 3 previous similar messages [ 8134.340846] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8147.218816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8147.286553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8147.392574] Lustre: 279674:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8147.667951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8151.214753] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8158.071959] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8161.938520] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8167.786950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8167.851676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8169.522919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8174.610950] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8182.627550] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8184.893736] Lustre: Mounted lustre-client [ 8186.015692] LustreError: 282177:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9b5f8b06b000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 8186.060538] Lustre: Unmounted lustre-client [ 8200.164457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8200.166828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8200.169892] Lustre: Skipped 3 previous similar messages [ 8200.177903] Lustre: Skipped 4 previous similar messages [ 8202.001534] Lustre: server umount lustre-OST0000 complete [ 8209.377958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8211.698326] Lustre: server umount lustre-MDT0000 complete [ 8215.609908] LustreError: 279664:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764799125 with bad export cookie 10063890276759897175 [ 8215.620109] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8215.638892] LustreError: 279664:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8216.305915] Lustre: server umount lustre-MDT0001 complete [ 8233.205090] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 8236.765391] Key type lgssc unregistered [ 8238.178349] LNet: 283457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8238.182780] LNetError: 283457:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8316.011216] LNet: Removed LNI 192.168.203.120@tcp [ 8316.727164] Key type .llcrypt unregistered [ 8316.729898] Key type ._llcrypt unregistered [ 8334.318915] Key type ._llcrypt registered [ 8334.320904] Key type .llcrypt registered [ 8334.461926] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 8349.159369] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 8349.859988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8349.918332] alg: No test for adler32 (adler32-zlib) [ 8350.984163] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 8351.228139] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 8352.906356] Key type lgssc registered [ 8354.072130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8363.203090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8370.072200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8375.577177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8381.810733] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8383.980394] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 17:01:31 (1764799291) [ 8391.283512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8391.358160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8391.376903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8392.613542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8392.639674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8392.728397] Lustre: lustre-MDT0000: new disk, initializing [ 8392.801659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8392.831356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8397.516281] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8408.660789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8408.767936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8408.888532] Lustre: 286662:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8409.007358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8409.012065] Lustre: Skipped 1 previous similar message [ 8409.059730] Lustre: lustre-MDT0001: new disk, initializing [ 8409.160864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8409.194584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8409.201218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8413.889423] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8422.886249] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8426.450951] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8433.316994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8433.378389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8433.601051] Lustre: lustre-OST0000: new disk, initializing [ 8433.605975] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8433.755525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8439.827480] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8441.470973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8441.483907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8441.662036] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8449.043399] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8462.307720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8462.313282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8462.329904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8462.350941] LustreError: Skipped 1 previous similar message [ 8465.064116] LustreError: 289105:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8465.333890] Lustre: server umount lustre-OST0000 complete [ 8470.507226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8470.509879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8470.514746] Lustre: Skipped 1 previous similar message [ 8470.548256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8470.551061] Lustre: Skipped 1 previous similar message [ 8472.544841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8472.548611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8472.596911] Lustre: Skipped 1 previous similar message [ 8477.670312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8477.688679] Lustre: Skipped 1 previous similar message [ 8482.791663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8482.795848] Lustre: Skipped 1 previous similar message [ 8484.319165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8484.395815] LustreError: 289307:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8484.415690] LustreError: 289307:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8484.563595] Lustre: server umount lustre-MDT0000 complete [ 8489.313719] LustreError: 286654:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764799399 with bad export cookie 4095076202301571815 [ 8489.318285] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8489.346796] LustreError: 286654:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8489.614685] LustreError: 289509:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8489.629905] LustreError: 289509:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8489.924064] Lustre: server umount lustre-MDT0001 complete [ 8503.177079] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 8506.640451] Key type lgssc unregistered [ 8506.989451] LNet: 290042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8507.001996] LNetError: 290042:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8507.042671] LNet: Removed LNI 192.168.203.120@tcp [ 8508.120447] Key type .llcrypt unregistered [ 8508.125839] Key type ._llcrypt unregistered [ 8606.536690] Key type ._llcrypt registered [ 8606.549422] Key type .llcrypt registered [ 8606.737349] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 8607.099472] Key type .llcrypt unregistered [ 8607.101063] Key type ._llcrypt unregistered [ 8687.975689] Key type ._llcrypt registered [ 8687.978362] Key type .llcrypt registered [ 8688.104657] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 8688.422379] Key type .llcrypt unregistered [ 8688.425228] Key type ._llcrypt unregistered [ 8716.251191] Key type ._llcrypt registered [ 8716.252815] Key type .llcrypt registered [ 8716.370206] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 8716.806777] Key type .llcrypt unregistered [ 8716.808669] Key type ._llcrypt unregistered [ 8791.326361] Key type ._llcrypt registered [ 8791.328014] Key type .llcrypt registered [ 8791.509338] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 8791.877290] Key type .llcrypt unregistered [ 8791.879912] Key type ._llcrypt unregistered [ 8821.305239] Key type ._llcrypt registered [ 8821.309866] Key type .llcrypt registered [ 8821.447057] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 8821.841477] Key type .llcrypt unregistered [ 8821.843802] Key type ._llcrypt unregistered [ 8852.830352] Key type ._llcrypt registered [ 8852.832200] Key type .llcrypt registered [ 8853.015345] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 8853.306885] Key type .llcrypt unregistered [ 8853.309267] Key type ._llcrypt unregistered [ 8924.226355] Key type ._llcrypt registered [ 8924.228490] Key type .llcrypt registered [ 8924.334349] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 8924.608457] Key type .llcrypt unregistered [ 8924.609983] Key type ._llcrypt unregistered [ 8952.497552] Key type ._llcrypt registered [ 8952.499015] Key type .llcrypt registered [ 8952.645547] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 8952.939067] Key type .llcrypt unregistered [ 8952.946417] Key type ._llcrypt unregistered [ 8983.452236] Key type ._llcrypt registered [ 8983.453639] Key type .llcrypt registered [ 8983.573279] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 8983.961577] Key type .llcrypt unregistered [ 8983.966285] Key type ._llcrypt unregistered [ 9040.622624] Key type ._llcrypt registered [ 9040.626755] Key type .llcrypt registered [ 9040.744491] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 9054.225254] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 9054.825076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9054.862824] alg: No test for adler32 (adler32-zlib) [ 9055.947621] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 9056.193641] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 9057.895201] Key type lgssc registered [ 9059.050794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9068.337490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9074.561743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9080.397936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9086.127770] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9087.796761] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8916 sec ========== 17:13:15 (1764799995) [ 9089.618870] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:13:17 (1764799997) === [ 9092.895332] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:13:20 (1764800000) === [ 9131.336265] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 9133.965911] Key type lgssc unregistered [ 9134.237679] LNet: 309172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9134.242946] LNetError: 309172:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9134.263139] LNet: Removed LNI 192.168.203.120@tcp [ 9134.976771] Key type .llcrypt unregistered [ 9134.978821] Key type ._llcrypt unregistered