[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 643902394 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001046] APIC: Switch to symmetric I/O mode setup [ 0.003636] x2apic enabled [ 0.004024] Switched APIC routing to physical x2apic. [ 0.005023] kvm-guest: setup PV IPIs [ 0.010000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010045] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011031] pid_max: default: 32768 minimum: 301 [ 0.012297] LSM: Security Framework initializing [ 0.013129] Yama: becoming mindful. [ 0.014182] SELinux: Initializing. [ 0.015115] *** VALIDATE selinux *** [ 0.032405] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.041658] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.043369] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.044340] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.046274] *** VALIDATE tmpfs *** [ 0.048277] *** VALIDATE proc *** [ 0.050171] *** VALIDATE cgroup *** [ 0.051011] *** VALIDATE cgroup2 *** [ 0.052433] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.054073] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.055021] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.056052] Spectre V2 : User space: Vulnerable [ 0.057021] Speculative Store Bypass: Vulnerable [ 0.062323] debug: unmapping init [mem 0xffffffffb3c59000-0xffffffffb3c60fff] [ 0.065752] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.067537] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.068029] ... version: 2 [ 0.069039] ... bit width: 48 [ 0.070025] ... generic registers: 4 [ 0.071031] ... value mask: 0000ffffffffffff [ 0.072017] ... max period: 00007fffffffffff [ 0.073021] ... fixed-purpose events: 3 [ 0.074019] ... event mask: 000000070000000f [ 0.075498] rcu: Hierarchical SRCU implementation. [ 0.077599] smp: Bringing up secondary CPUs ... [ 0.079277] x86: Booting SMP configuration: [ 0.080018] .... node #0, CPUs: #1 #2 #3 [ 0.093485] smp: Brought up 1 node, 4 CPUs [ 0.095011] smpboot: Max logical packages: 1 [ 0.096027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.154055] node 0 deferred pages initialised in 55ms [ 0.159110] devtmpfs: initialized [ 0.160288] x86/mm: Memory block size: 128MB [ 0.163000] gcov: version magic: 0x41383552 [ 0.168293] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.183133] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.189166] pinctrl core: initialized pinctrl subsystem [ 0.199048] [ 0.201023] ************************************************************* [ 0.210052] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.222025] ** ** [ 0.236376] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.248850] ** ** [ 0.254285] ** This means that this kernel is built to expose internal ** [ 0.263081] ** IOMMU data structures, which may compromise security on ** [ 0.268029] ** your system. ** [ 0.272023] ** ** [ 0.276100] ** If you see this message and you are not debugging the ** [ 0.279049] ** kernel, report this immediately to your vendor! ** [ 0.282010] ** ** [ 0.294030] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.303041] ************************************************************* [ 0.320865] NET: Registered protocol family 16 [ 0.326939] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.336103] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.344261] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.358096] cpuidle: using governor menu [ 0.360832] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.361639] PCI: Using configuration type 1 for base access [ 0.364224] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.372159] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.374124] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.379031] cryptd: max_cpu_qlen set to 1000 [ 0.382315] ACPI: Added _OSI(Module Device) [ 0.383000] ACPI: Added _OSI(Processor Device) [ 0.383000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.385079] ACPI: Added _OSI(Processor Aggregator Device) [ 0.391956] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.399839] ACPI: Interpreter enabled [ 0.401157] ACPI: PM: (supports S0 S3 S4 S5) [ 0.403014] ACPI: Using IOAPIC for interrupt routing [ 0.406133] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.409398] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.422272] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.425048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.426012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.430127] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.436682] acpiphp: Slot [2] registered [ 0.438391] acpiphp: Slot [5] registered [ 0.440276] acpiphp: Slot [6] registered [ 0.442123] acpiphp: Slot [7] registered [ 0.444112] acpiphp: Slot [8] registered [ 0.446107] acpiphp: Slot [9] registered [ 0.447104] acpiphp: Slot [10] registered [ 0.448259] acpiphp: Slot [3] registered [ 0.450081] acpiphp: Slot [4] registered [ 0.451000] acpiphp: Slot [11] registered [ 0.451000] acpiphp: Slot [12] registered [ 0.451000] acpiphp: Slot [13] registered [ 0.452075] acpiphp: Slot [14] registered [ 0.453066] acpiphp: Slot [15] registered [ 0.454072] acpiphp: Slot [16] registered [ 0.455028] acpiphp: Slot [17] registered [ 0.456036] acpiphp: Slot [18] registered [ 0.457270] acpiphp: Slot [19] registered [ 0.459092] acpiphp: Slot [20] registered [ 0.461122] acpiphp: Slot [21] registered [ 0.463113] acpiphp: Slot [22] registered [ 0.465080] acpiphp: Slot [23] registered [ 0.467101] acpiphp: Slot [24] registered [ 0.469100] acpiphp: Slot [25] registered [ 0.470086] acpiphp: Slot [26] registered [ 0.471053] acpiphp: Slot [27] registered [ 0.472053] acpiphp: Slot [28] registered [ 0.473068] acpiphp: Slot [29] registered [ 0.474017] acpiphp: Slot [30] registered [ 0.475056] acpiphp: Slot [31] registered [ 0.476036] PCI host bridge to bus 0000:00 [ 0.478015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.480043] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.481062] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.483020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.486023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.490024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.495150] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.502007] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.508707] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.526480] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.530577] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.533022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.535016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.538015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.540727] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.542717] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.545047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.547875] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.552012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.565013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.568014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.573607] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.584021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.592022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.609018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.620253] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.626020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.634023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.679020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.713808] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.732019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.738015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.752020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.765572] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.775021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.788018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.805017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.833838] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.839014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.845015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.860028] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.872350] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.878015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.886016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.901017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.913509] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.916413] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.919393] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.922399] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.925212] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.931000] iommu: Default domain type: Passthrough [ 0.932414] SCSI subsystem initialized [ 0.934131] ACPI: bus type USB registered [ 0.936098] usbcore: registered new interface driver usbfs [ 0.939116] usbcore: registered new interface driver hub [ 0.941073] usbcore: registered new device driver usb [ 0.943157] pps_core: LinuxPPS API ver. 1 registered [ 0.946010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.950168] PTP clock support registered [ 0.952208] EDAC MC: Ver: 3.0.0 [ 0.954087] PCI: Using ACPI for IRQ routing [ 0.955000] NetLabel: Initializing [ 0.957011] NetLabel: domain hash size = 128 [ 0.959011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.963144] NetLabel: unlabeled traffic allowed by default [ 0.966004] vgaarb: loaded [ 0.968341] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.972017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.981171] clocksource: Switched to clocksource kvm-clock [ 1.116974] VFS: Disk quotas dquot_6.6.0 [ 1.119712] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.123212] *** VALIDATE ramfs *** [ 1.124703] *** VALIDATE hugetlbfs *** [ 1.126699] pnp: PnP ACPI init [ 1.129583] pnp: PnP ACPI: found 6 devices [ 1.147129] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.151342] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.154178] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.157019] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.160251] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.163049] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.166610] NET: Registered protocol family 2 [ 1.169676] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.175424] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.179236] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.185705] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.203830] TCP: Hash tables configured (established 65536 bind 65536) [ 1.207992] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.212752] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.216640] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.221541] NET: Registered protocol family 1 [ 1.224894] RPC: Registered named UNIX socket transport module. [ 1.227626] RPC: Registered udp transport module. [ 1.229720] RPC: Registered tcp transport module. [ 1.231402] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.234647] NET: Registered protocol family 44 [ 1.236311] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.239251] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.241891] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.244648] PCI: CLS 0 bytes, default 64 [ 1.247174] Unpacking initramfs... [ 3.045928] debug: unmapping init [mem 0xffff93cf7cc54000-0xffff93cf7ffbffff] [ 3.059706] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.063684] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.067504] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.671965] Initialise system trusted keyrings [ 3.674691] Key type blacklist registered [ 3.677413] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.690151] zbud: loaded [ 3.694072] *** VALIDATE nfs *** [ 3.695923] *** VALIDATE nfs4 *** [ 3.698766] pstore: using deflate compression [ 3.705562] Platform Keyring initialized [ 3.889274] NET: Registered protocol family 38 [ 3.891934] Key type asymmetric registered [ 3.894213] Asymmetric key parser 'x509' registered [ 3.896605] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.900816] io scheduler mq-deadline registered [ 3.903316] io scheduler kyber registered [ 3.905686] io scheduler bfq registered [ 3.908742] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.912910] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.916775] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.920377] ACPI: Power Button [PWRF] [ 3.927099] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.935790] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.950410] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.957549] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.970863] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.996490] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.024543] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.031293] Non-volatile memory driver v1.3 [ 4.033840] Linux agpgart interface v0.103 [ 4.070259] virtio_blk virtio1: [vda] 134032 512-byte logical blocks (68.6 MB/65.4 MiB) [ 4.072519] vda: detected capacity change from 0 to 68624384 [ 4.171834] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.175491] vdb: detected capacity change from 0 to 1073741824 [ 4.291613] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.295869] vdc: detected capacity change from 0 to 2621440000 [ 4.312771] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.316562] vdd: detected capacity change from 0 to 2621440000 [ 4.337208] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.340883] vde: detected capacity change from 0 to 4294967296 [ 4.387609] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.395474] vdf: detected capacity change from 0 to 4294967296 [ 4.416734] libphy: Fixed MDIO Bus: probed [ 4.432667] usbcore: registered new interface driver usbserial_generic [ 4.438862] usbserial: USB Serial support registered for generic [ 4.445645] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.458297] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.463876] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.470399] mousedev: PS/2 mouse device common for all mice [ 4.477321] rtc_cmos 00:05: RTC can wake from S4 [ 4.480711] rtc_cmos 00:05: registered as rtc0 [ 4.482871] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.486713] intel_pstate: CPU model not supported [ 4.489818] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.499204] hid: raw HID events driver (C) Jiri Kosina [ 4.505646] usbcore: registered new interface driver usbhid [ 4.514301] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.517506] usbhid: USB HID core driver [ 4.532932] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.546693] drop_monitor: Initializing network drop monitor service [ 4.551595] Initializing XFRM netlink socket [ 4.556200] NET: Registered protocol family 10 [ 4.562311] Segment Routing with IPv6 [ 4.564352] NET: Registered protocol family 17 [ 4.567572] mpls_gso: MPLS GSO support [ 4.575893] RAS: Correctable Errors collector initialized. [ 4.579373] AVX version of gcm_enc/dec engaged. [ 4.582192] AES CTR mode by8 optimization enabled [ 4.934314] sched_clock: Marking stable (4934198883, 0)->(6093550635, -1159351752) [ 4.940192] registered taskstats version 1 [ 4.942728] Loading compiled-in X.509 certificates [ 4.945800] zswap: loaded using pool lzo/zbud [ 4.976673] Key type big_key registered [ 4.992930] Key type encrypted registered [ 4.995537] ima: No TPM chip found, activating TPM-bypass! [ 4.999984] ima: Allocated hash algorithm: sha1 [ 5.005640] ima: No architecture policies found [ 5.010966] evm: Initialising EVM extended attributes: [ 5.016901] evm: security.selinux [ 5.019866] evm: security.ima [ 5.022885] evm: security.capability [ 5.027614] evm: HMAC attrs: 0x1 [ 5.033617] rtc_cmos 00:05: setting system clock to 2026-01-22 09:10:12 UTC (1769073012) [ 5.051970] debug: unmapping init [mem 0xffffffffb4c03000-0xffffffffb4dfffff] [ 5.061889] debug: unmapping init [mem 0xffffffffb3982000-0xffffffffb3c58fff] [ 5.075196] Write protecting the kernel read-only data: 28672k [ 5.085383] debug: unmapping init [mem 0xffffffffb2003000-0xffffffffb21fffff] [ 5.092104] debug: unmapping init [mem 0xffffffffb2914000-0xffffffffb29fffff] [ 5.483843] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.524890] systemd[1]: Detected virtualization kvm. [ 5.538595] systemd[1]: Detected architecture x86-64. [ 5.553265] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.735360] systemd[1]: No hostname configured. [ 5.737724] systemd[1]: Set hostname to . [ 5.740334] random: systemd: uninitialized urandom read (16 bytes read) [ 5.743488] systemd[1]: Initializing machine ID from random generator. [ 5.813355] random: ln: uninitialized urandom read (6 bytes read) [ 5.969668] random: systemd: uninitialized urandom read (16 bytes read) [ 5.973371] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.979645] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.992802] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.298681] device-mapper: uevent: version 1.0.3 [ 8.301597] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook...[ 11.462907] virtio_net virtio0 ens2: renamed from eth0 [ 11.504157] random: fast init done [ 12.004336] scsi host0: ata_piix [ 12.038262] scsi host1: ata_piix [ 12.039878] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.042693] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.668086] random: crng init done [ 17.669441] random: 7 urandom warning(s) missed due to ratelimiting [ 20.853594] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.791529] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.453177] printk: systemd: 25 output lines suppressed due to ratelimiting [ 25.215861] SELinux: Disabled at runtime. [ 25.290692] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.300215] systemd[1]: Detected virtualization kvm. [ 25.301744] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.385940] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.389771] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.411483] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.417273] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.423906] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.435394] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.450931] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. [ 26.959807] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 29.015962] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.815484] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 32.373111] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 34.257524] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 34.429886] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit)[ 38.673676] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit)[ 39.647163] NFS: Registering the id_resolver key type [ 39.648934] Key type id_resolver registered [ 39.650640] Key type id_legacy registered [ *] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg351-server login: [ 85.392311] libcfs: loading out-of-tree module taints kernel. [ 85.431935] Key type ._llcrypt registered [ 85.433304] Key type .llcrypt registered [ 85.503948] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 97.252119] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 98.332966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 98.342523] alg: No test for adler32 (adler32-zlib) [ 99.511291] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 99.978760] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 101.647723] Key type lgssc registered [ 102.709430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 114.556733] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 115.474122] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 121.646271] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 127.102408] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 132.633521] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 143.013646] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 152.014463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 152.057928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 152.071098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.271397] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 153.298557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 153.376553] Lustre: lustre-MDT0000: new disk, initializing [ 153.467624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 153.489259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 156.499735] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 166.231527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 166.307091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 166.354500] Lustre: 6489:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 166.370635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 166.372747] Lustre: Skipped 1 previous similar message [ 166.453876] Lustre: lustre-MDT0001: new disk, initializing [ 166.520791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 166.539913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 166.553520] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 169.061951] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.259386] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 178.328419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 178.376243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 178.542675] Lustre: lustre-OST0000: new disk, initializing [ 178.544984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 178.615066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.670115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 179.686108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 179.714930] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 182.094442] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.094888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 192.149850] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 192.212149] Lustre: lustre-OST0001: new disk, initializing [ 192.222180] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 192.277113] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 196.924708] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.214205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 200.219811] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 200.247259] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 205.985470] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 212.090673] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 220.116045] hrtimer: interrupt took 6037622 ns [ 221.102676] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing check_logdir /tmp/testlogs/ [ 223.721796] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing yml_node [ 226.587834] Lustre: DEBUG MARKER: Client: 2.17.50.4 [ 228.180209] Lustre: DEBUG MARKER: MDS: 2.17.50.4 [ 229.924214] Lustre: DEBUG MARKER: OSS: 2.17.50.4 [ 231.143778] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jan 22 04:13:58 EST 2026 [ 243.526835] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 244.513258] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 258.528678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 258.537088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 258.557660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 261.600348] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 261.600566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 261.610549] Lustre: Skipped 2 previous similar messages [ 261.613444] Lustre: Skipped 2 previous similar messages [ 263.705355] LustreError: 12190:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 263.816172] Lustre: server umount lustre-MDT0000 complete [ 266.342714] LustreError: 6482:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769073273 with bad export cookie 11763789425056381357 [ 266.344335] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 266.351489] LustreError: 6482:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 266.720599] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 266.747050] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 266.752670] Lustre: Skipped 2 previous similar messages [ 271.842433] LustreError: 6498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 271.843529] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 271.857239] LustreError: 6498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 271.868544] Lustre: Skipped 1 previous similar message [ 272.535699] LustreError: 12393:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 272.542622] LustreError: 12393:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 272.665883] Lustre: server umount lustre-MDT0001 complete [ 274.902300] LustreError: 12594:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 274.905597] LustreError: 12594:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 274.936788] Lustre: server umount lustre-OST0000 complete [ 277.012994] LustreError: 12794:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 277.019036] LustreError: 12794:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 277.110751] Lustre: server umount lustre-OST0001 complete [ 280.534622] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 286.226357] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 292.388911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 300.541734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 311.404888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 321.987065] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 333.760384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 333.910582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 334.215786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 334.253508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 334.380357] Lustre: lustre-MDT0000: new disk, initializing [ 334.485692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 334.524216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 343.598400] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 358.113139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 358.223773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 358.379267] Lustre: 14979:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 358.455920] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 358.460107] Lustre: Skipped 1 previous similar message [ 358.602682] Lustre: lustre-MDT0001: new disk, initializing [ 358.786833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 358.891844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 358.917481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 366.095809] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 375.195406] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 378.691465] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 386.064717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 386.153311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 386.607841] Lustre: lustre-OST0000: new disk, initializing [ 386.611328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 386.770585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.432429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 388.474990] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 388.733299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 394.298662] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 403.617872] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 409.121779] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 409.460523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 414.102824] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 414.347883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 419.296204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 419.298945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 419.299567] LustreError: Skipped 1 previous similar message [ 419.308434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 419.325923] Lustre: Skipped 2 previous similar messages [ 422.342381] LustreError: 17926:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 422.349255] LustreError: 17926:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 422.595935] Lustre: server umount lustre-OST0000 complete [ 429.548622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 429.551887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 429.578898] Lustre: Skipped 1 previous similar message [ 429.592393] Lustre: Skipped 2 previous similar messages [ 432.440747] LustreError: 18127:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 432.455475] LustreError: 18127:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 432.737479] Lustre: server umount lustre-MDT0000 complete [ 434.659645] LustreError: 14986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 434.691424] LustreError: 14986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 436.546722] LustreError: 16733:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769073444 with bad export cookie 11763789425056382897 [ 436.550366] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 436.558589] LustreError: 16733:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 437.020756] Lustre: server umount lustre-MDT0001 complete [ 442.789753] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:17:29 (1769073449) [ 462.020310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.358389] LustreError: 19463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 462.427677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.428042] LustreError: 19463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 467.802679] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 472.550484] LustreError: 19463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.997551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 485.039571] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 492.163772] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 496.321699] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 498.668681] Lustre: Failing over lustre-MDT0000 [ 498.791397] LustreError: 20807:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 498.796948] LustreError: 20807:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 499.028992] Lustre: server umount lustre-MDT0000 complete [ 499.185502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 499.204931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 499.244830] LustreError: 19463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.270059] LustreError: 19463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 518.642858] LustreError: 19464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.666191] LustreError: 19464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 519.664450] Lustre: 3636:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769073510/real 1769073510] req@ffff93cec50f7480 x1855007602391296/t0(0) o400->MGC192.168.203.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769073526 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 519.744143] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 521.911343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.016490] LustreError: 3632:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff93cec50e4e00 x1855007602394112/t0(0) o250->MGC192.168.203.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 530.369150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 530.383100] Lustre: Skipped 1 previous similar message [ 535.081447] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 535.522877] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 535.533630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 535.573603] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 543.573946] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 548.261122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.267709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 548.283185] Lustre: Skipped 2 previous similar messages [ 548.293939] Lustre: Skipped 1 previous similar message [ 553.511874] LustreError: 21788:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 553.522730] LustreError: 21788:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 553.602228] Lustre: server umount lustre-MDT0000 complete [ 557.357552] LustreError: 19446:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769073564 with bad export cookie 11763789425056384311 [ 557.363571] LustreError: 19446:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 557.364886] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 557.637326] Lustre: server umount lustre-MDT0001 complete [ 566.673692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.041798] LustreError: 22470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.067424] LustreError: 22470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 571.634722] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 580.822464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 586.999588] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 593.646518] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 599.701722] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 602.862631] Lustre: Failing over lustre-MDT0000 [ 603.381174] Lustre: server umount lustre-MDT0000 complete [ 605.985458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 606.021207] Lustre: Skipped 1 previous similar message [ 622.495159] Lustre: 3636:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769073613/real 1769073613] req@ffff93cfff619180 x1855007602420608/t0(0) o400->MGC192.168.203.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769073629 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 622.527720] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 624.202696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 632.800228] LustreError: 22470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 632.813702] LustreError: 22470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 633.111777] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 638.443893] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 638.449487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 638.476609] Lustre: Skipped 1 previous similar message [ 638.512804] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 638.570937] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 647.823167] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 653.797250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 653.810478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 653.815811] Lustre: Skipped 2 previous similar messages [ 657.952157] LustreError: 24794:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 657.962778] LustreError: 24794:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 658.171933] Lustre: server umount lustre-MDT0000 complete [ 662.192862] LustreError: 22449:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769073669 with bad export cookie 11763789425056385242 [ 662.212294] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 662.220912] LustreError: 22449:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 662.693005] Lustre: server umount lustre-MDT0001 complete [ 674.149333] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 04:21:19 (1769073679) [ 693.530157] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 705.182107] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 717.002946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 723.800056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 730.551408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 737.829483] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 749.062745] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 759.735881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 759.793136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 759.973627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 760.048486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 760.154358] Lustre: lustre-MDT0000: new disk, initializing [ 760.269256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 760.273905] Lustre: Skipped 3 previous similar messages [ 760.299675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 765.219689] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 776.908499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 777.008614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 777.157375] Lustre: 28470:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 777.690137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 782.484735] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 791.585382] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 796.913944] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 806.132293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 806.286074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 808.589037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 808.636736] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 814.064874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 814.092665] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 814.101606] Lustre: Skipped 1 previous similar message [ 814.107879] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 815.198665] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 824.237883] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 830.635150] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 860.134627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 860.136476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 860.137054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 860.137060] Lustre: Skipped 2 previous similar messages [ 860.195524] Lustre: Skipped 3 previous similar messages [ 870.367335] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 870.543608] LustreError: 31135:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 870.549523] LustreError: 31135:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 870.853477] Lustre: server umount lustre-OST0000 complete [ 881.863578] Lustre: server umount lustre-MDT0000 complete [ 885.728315] LustreError: 29266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 885.742899] LustreError: 29266:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 886.346690] LustreError: 28465:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769073893 with bad export cookie 11763789425056385641 [ 886.357117] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 886.378832] LustreError: 28465:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 887.389706] Lustre: server umount lustre-MDT0001 complete [ 902.450315] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 905.474940] Key type lgssc unregistered [ 905.761248] LNet: 32074:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.775605] LNetError: 32074:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.805603] LNet: Removed LNI 192.168.203.151@tcp [ 906.574140] Key type .llcrypt unregistered [ 906.576091] Key type ._llcrypt unregistered [ 933.513770] Key type ._llcrypt registered [ 933.515886] Key type .llcrypt registered [ 933.608510] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 935.041956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 935.089575] alg: No test for adler32 (adler32-zlib) [ 936.262059] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 936.581184] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 938.442522] Key type lgssc registered [ 939.805604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 948.536763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 948.567980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 950.227224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.774515] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 960.484411] LustreError: 33471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 960.507296] LustreError: 33471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 964.983841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 965.487627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 970.890594] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 977.602648] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 982.212466] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 989.950692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 990.187902] LustreError: 35074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 990.349849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 995.517753] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 997.918544] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1005.272301] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1020.242645] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 04:27:06 (1769074026) [ 1022.464413] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1024.755442] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1033.397582] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 04:27:19 (1769074039) [ 1044.762316] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 04:27:30 (1769074050) [ 1046.306055] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1058.611962] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1058.619502] Lustre: Skipped 2 previous similar messages [ 1060.946408] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 04:27:47 (1769074067) [ 1063.336629] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1063.349693] Lustre: Skipped 2 previous similar messages [ 1072.657333] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1072.663990] Lustre: Skipped 4 previous similar messages [ 1089.950619] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 04:28:16 (1769074096) [ 1093.639931] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1097.947963] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1109.282722] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 04:28:35 (1769074115) [ 1112.028653] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1112.035890] Lustre: Skipped 6 previous similar messages [ 1129.007795] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 04:28:55 (1769074135) [ 1149.420361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1149.423256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.472048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1149.478269] Lustre: Skipped 1 previous similar message [ 1151.424510] LustreError: 40057:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1151.583055] Lustre: server umount lustre-OST0000 complete [ 1159.649722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.658508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1159.664109] Lustre: Skipped 2 previous similar messages [ 1159.684421] Lustre: Skipped 1 previous similar message [ 1161.646230] LustreError: 40259:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1161.649994] LustreError: 40259:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1161.848872] Lustre: server umount lustre-MDT0000 complete [ 1164.781406] LustreError: 33471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.802791] LustreError: 33471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1165.401621] LustreError: 37558:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074172 with bad export cookie 11934050741628622020 [ 1165.413788] LustreError: 37558:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1165.422525] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1165.576041] LustreError: 40460:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1165.578978] LustreError: 40460:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1165.876821] Lustre: server umount lustre-MDT0001 complete [ 1178.414333] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1181.989208] Key type lgssc unregistered [ 1182.354122] LNet: 40993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.365357] LNetError: 40993:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.418105] LNet: Removed LNI 192.168.203.151@tcp [ 1183.467039] Key type .llcrypt unregistered [ 1183.469529] Key type ._llcrypt unregistered [ 1203.946612] Key type ._llcrypt registered [ 1203.949731] Key type .llcrypt registered [ 1204.059365] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 1221.856801] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1223.264985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1223.354938] alg: No test for adler32 (adler32-zlib) [ 1224.499876] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 1224.836342] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1226.521907] Key type lgssc registered [ 1227.645979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1236.260778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1243.489866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1250.281924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1257.623516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1264.983920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1265.036216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1265.048468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1266.385943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1266.436388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1266.544545] Lustre: lustre-MDT0000: new disk, initializing [ 1266.636948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1266.669328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1270.863822] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1280.386274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1280.465138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1280.563415] Lustre: 44122:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1280.615672] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1280.622311] Lustre: Skipped 1 previous similar message [ 1280.705372] Lustre: lustre-MDT0001: new disk, initializing [ 1280.795161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1280.834319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1280.852731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1284.594752] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1292.298725] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1296.043864] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1303.010651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1303.088095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1303.355666] Lustre: lustre-OST0000: new disk, initializing [ 1303.360228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1303.491701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1309.177740] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1312.892134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1312.911351] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1313.080089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1316.425478] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1320.707268] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1320.858571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1325.528757] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1325.726859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1328.611724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1328.617290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1328.618328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1328.633413] Lustre: Skipped 1 previous similar message [ 1328.664305] LustreError: Skipped 1 previous similar message [ 1333.733605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1333.738371] Lustre: Skipped 2 previous similar messages [ 1333.830124] LustreError: 47063:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1333.943518] Lustre: server umount lustre-OST0000 complete [ 1338.852880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1338.876349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1338.897395] Lustre: Skipped 1 previous similar message [ 1343.968504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1343.979825] Lustre: Skipped 1 previous similar message [ 1349.089128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1352.163552] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1352.349996] LustreError: 47265:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1352.360544] LustreError: 47265:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1352.542871] Lustre: server umount lustre-MDT0000 complete [ 1356.835807] LustreError: 44114:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074364 with bad export cookie 4791589495504332371 [ 1356.851220] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1357.120034] LustreError: 47469:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1357.127353] LustreError: 47469:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1357.605615] Lustre: server umount lustre-MDT0001 complete [ 1366.954984] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 04:32:53 (1769074373) [ 1374.761444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1375.240523] LustreError: 48189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1375.270520] LustreError: 48189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1375.327449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1379.803955] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1380.325992] LustreError: 48189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1385.445373] LustreError: 48189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1388.385526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1388.738993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1393.923730] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1399.544781] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1403.426736] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1410.648364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1411.187846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1418.062404] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1423.895741] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1428.548457] LustreError: 50386:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1429.561442] Lustre: Modifying parameter general.timeout=1 in log params [ 1430.958228] Lustre: Modifying parameter general.timeout=2 in log params [ 1432.978740] Lustre: Modifying parameter general.timeout=4 in log params [ 1432.981961] Lustre: Skipped 1 previous similar message [ 1437.699120] Lustre: Modifying parameter general.timeout=9 in log params [ 1437.706225] Lustre: Skipped 4 previous similar messages [ 1446.457708] Lustre: Modifying parameter general.timeout=17 in log params [ 1446.466912] Lustre: Skipped 7 previous similar messages [ 1463.193329] Lustre: Modifying parameter general.timeout=34 in log params [ 1463.196266] Lustre: Skipped 16 previous similar messages [ 1495.977394] Lustre: Modifying parameter general.timeout=66 in log params [ 1495.987508] Lustre: Skipped 31 previous similar messages [ 1564.333387] Lustre: Disabling parameter general.timeout= in log params [ 1564.340318] Lustre: Skipped 63 previous similar messages [ 1567.718380] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 04:36:14 (1769074574) [ 1570.936798] Lustre: 57343:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1572.969195] Lustre: 57441:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1577.150936] Lustre: 57654:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1577.161402] Lustre: 57654:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1581.478930] Lustre: 57851:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1581.486980] Lustre: 57851:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1612.038305] Lustre: 59205:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1612.045582] LustreError: 59205:0:(mgs_handler.c:1129:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1616.801391] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 04:37:03 (1769074623) [ 1634.275791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1634.287031] Lustre: Skipped 3 previous similar messages [ 1634.304837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1634.312045] Lustre: Skipped 3 previous similar messages [ 1636.422211] LustreError: 59905:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1636.441692] LustreError: 59905:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1636.567466] Lustre: server umount lustre-MDT0000 complete [ 1639.393259] LustreError: 48902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1639.413539] LustreError: 48902:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1640.510079] LustreError: 48168:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074647 with bad export cookie 4791589495504333260 [ 1640.511662] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1640.521411] LustreError: 48168:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1640.686258] LustreError: 60108:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1640.692632] LustreError: 60108:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1640.932901] Lustre: server umount lustre-MDT0001 complete [ 1655.580963] LustreError: 60309:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1655.602421] LustreError: 60309:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1655.882283] Lustre: server umount lustre-OST0000 complete [ 1665.151001] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1669.588198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1673.781042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1678.254739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1686.647173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1686.965917] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1687.391606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1692.320844] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1701.822358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1701.978529] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1702.042165] Lustre: 61763:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1707.718565] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1713.674760] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1718.270174] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1726.220782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1726.357076] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1726.554110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1726.559771] Lustre: Skipped 1 previous similar message [ 1732.889401] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1738.554050] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1740.498527] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1740.510365] Lustre: Skipped 23 previous similar messages [ 1740.705763] Lustre: 63999:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1750.503131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1750.509627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1750.528324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1754.443535] LustreError: 64254:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1754.450523] LustreError: 64254:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1754.659627] Lustre: server umount lustre-OST0000 complete [ 1758.701605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1758.704734] LustreError: Skipped 1 previous similar message [ 1758.724886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1758.752865] Lustre: Skipped 1 previous similar message [ 1765.352783] Lustre: server umount lustre-MDT0000 complete [ 1766.370870] LustreError: 61774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1766.405956] LustreError: 61774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1770.113035] LustreError: 61756:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074777 with bad export cookie 4791589495504336340 [ 1770.119520] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1770.127356] LustreError: 61756:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1770.597727] Lustre: server umount lustre-MDT0001 complete [ 1785.318698] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1788.733393] Key type lgssc unregistered [ 1789.008413] LNet: 65192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1789.023028] LNetError: 65192:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1789.050984] LNet: Removed LNI 192.168.203.151@tcp [ 1789.865327] Key type .llcrypt unregistered [ 1789.866972] Key type ._llcrypt unregistered [ 1815.137587] Key type ._llcrypt registered [ 1815.139741] Key type .llcrypt registered [ 1815.252883] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1816.471817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1816.575993] alg: No test for adler32 (adler32-zlib) [ 1817.579560] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 1817.837786] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1819.615162] Key type lgssc registered [ 1820.673314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1828.775797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1828.803953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.339769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.361491] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1839.591848] LustreError: 66587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1839.615850] LustreError: 66587:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1842.544993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1843.391798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1847.933046] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1853.439386] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1857.934963] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1865.064911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1865.560966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1872.089615] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1878.255767] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1887.034867] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1899.488718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1899.499892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1899.516688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1900.515650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1900.526127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.550115] Lustre: Skipped 1 previous similar message [ 1902.077519] LustreError: 69298:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1902.194968] Lustre: server umount lustre-MDT0000 complete [ 1905.637238] LustreError: 66587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1905.660665] LustreError: 66587:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1906.847539] LustreError: 66567:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074914 with bad export cookie 5472359816592173086 [ 1906.856594] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1907.109348] LustreError: 69500:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1907.113336] LustreError: 69500:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1907.460860] Lustre: server umount lustre-MDT0001 complete [ 1922.188991] LustreError: 69700:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1922.198156] LustreError: 69700:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1922.449389] Lustre: server umount lustre-OST0000 complete [ 1931.603965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1935.544804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1939.627740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1944.358615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1951.351312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.730427] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1951.796995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1952.088902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1956.032344] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1964.447945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1964.518125] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1964.539353] Lustre: 71154:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1964.893624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1969.019693] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1975.082901] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1979.814546] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1987.963729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1988.302658] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1988.970222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.909782] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2001.660394] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2003.529714] Lustre: 73395:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2005.287232] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2009.575206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2009.577322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.590365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2009.590373] Lustre: Skipped 2 previous similar messages [ 2009.625593] Lustre: Skipped 1 previous similar message [ 2014.689484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2014.698883] Lustre: Skipped 2 previous similar messages [ 2015.422712] LustreError: 73645:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2015.430908] LustreError: 73645:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2015.623893] Lustre: server umount lustre-OST0000 complete [ 2021.352609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2021.363525] LustreError: Skipped 1 previous similar message [ 2021.370555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2021.389647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2026.344296] LustreError: 73853:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2026.356526] LustreError: 73853:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2026.468029] Lustre: server umount lustre-MDT0000 complete [ 2030.049736] LustreError: 71983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2030.086561] LustreError: 71983:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2031.060559] LustreError: 73397:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075038 with bad export cookie 5472359816592173933 [ 2031.068277] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2031.076781] LustreError: 73397:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2031.373849] Lustre: server umount lustre-MDT0001 complete [ 2044.890963] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 2047.477596] Key type lgssc unregistered [ 2047.764955] LNet: 74587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2047.777639] LNetError: 74587:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2047.794572] LNet: Removed LNI 192.168.203.151@tcp [ 2048.547741] Key type .llcrypt unregistered [ 2048.552639] Key type ._llcrypt unregistered [ 2070.040908] Key type ._llcrypt registered [ 2070.043641] Key type .llcrypt registered [ 2070.198792] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2071.269259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2071.397385] alg: No test for adler32 (adler32-zlib) [ 2072.459748] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 2072.628718] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 2074.304336] Key type lgssc registered [ 2075.234717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2082.916723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2082.958864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.489667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2089.289786] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2094.051310] LustreError: 75960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2094.086049] LustreError: 75960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2099.168449] LustreError: 75960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2100.156495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2100.821369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2104.854986] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2110.510365] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2114.441774] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2121.471298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2122.211304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2129.318705] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2135.425444] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2142.330290] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2143.235982] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2145.049929] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2145.062447] Lustre: Skipped 1 previous similar message [ 2147.337594] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2147.342053] Lustre: Skipped 2 previous similar messages [ 2151.491423] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2151.504548] Lustre: Skipped 4 previous similar messages [ 2160.163739] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2160.166526] Lustre: Skipped 8 previous similar messages [ 2176.677721] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2176.681519] Lustre: Skipped 18 previous similar messages [ 2208.707716] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2208.711545] Lustre: Skipped 33 previous similar messages [ 2241.160748] Lustre: 83733:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2244.695030] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 04:47:30 (1769075250) [ 2246.359444] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2249.037803] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 04:47:35 (1769075255) [ 2257.381846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2257.390498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.410896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2257.443857] Lustre: Skipped 1 previous similar message [ 2259.538867] LustreError: 84030:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2259.867863] Lustre: server umount lustre-OST0000 complete [ 2264.546615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2264.555256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.572073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2264.594334] Lustre: Skipped 1 previous similar message [ 2267.617773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2267.622880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.738903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2272.750845] Lustre: Skipped 2 previous similar messages [ 2276.832846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2276.842756] Lustre: Skipped 1 previous similar message [ 2278.367570] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2278.451614] LustreError: 84231:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2278.458509] LustreError: 84231:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2278.604967] Lustre: server umount lustre-MDT0000 complete [ 2281.958655] LustreError: 75956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.999603] LustreError: 75956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2282.222091] LustreError: 78264:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075289 with bad export cookie 6399491663579821643 [ 2282.230160] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2282.231035] LustreError: 78264:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2282.433228] LustreError: 84434:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2282.438502] LustreError: 84434:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2282.614569] Lustre: server umount lustre-MDT0001 complete [ 2294.042262] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 2297.087514] Key type lgssc unregistered [ 2297.577815] LNet: 84966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2297.585443] LNetError: 84966:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2297.608514] LNet: Removed LNI 192.168.203.151@tcp [ 2298.454440] Key type .llcrypt unregistered [ 2298.457431] Key type ._llcrypt unregistered [ 2303.192185] Key type ._llcrypt registered [ 2303.195676] Key type .llcrypt registered [ 2303.330648] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2307.484461] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules [ 2308.565965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2308.630563] alg: No test for adler32 (adler32-zlib) [ 2309.928086] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 2310.173239] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 2311.945349] Key type lgssc registered [ 2313.043329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2314.395749] LustreError: 86131:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 2333.563591] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2336.654863] LustreError: 86131:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2343.749515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2343.773539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.227584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2348.609145] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2354.660931] LustreError: 87131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.680334] LustreError: 87131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2356.980505] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 04:49:23 (1769075363) [ 2359.795609] LustreError: 87131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.970234] LustreError: 87163:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2361.979793] LustreError: 87163:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2361.987586] LustreError: 87163:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2362.107466] LustreError: 87936:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2362.411792] Lustre: server umount lustre-MDT0000 complete [ 2377.130724] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 2379.696262] Key type lgssc unregistered [ 2380.006985] LNet: 88622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2380.018588] LNetError: 88622:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2380.031786] LNet: Removed LNI 192.168.203.151@tcp [ 2380.921889] Key type .llcrypt unregistered [ 2380.929781] Key type ._llcrypt unregistered [ 2400.281426] Key type ._llcrypt registered [ 2400.284223] Key type .llcrypt registered [ 2400.402113] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2401.447232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2401.483826] alg: No test for adler32 (adler32-zlib) [ 2402.714873] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 2403.088531] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 2404.855254] Key type lgssc registered [ 2406.184398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2414.132802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2414.150431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.804341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2420.043903] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2426.358839] LustreError: 89820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.380828] LustreError: 89820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2428.997206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2429.760531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2434.648786] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2440.690359] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2444.440463] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2452.893455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2453.408782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2460.184090] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2467.682565] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2516.148678] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 04:52:03 (1769075523) [ 2517.359152] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2519.187860] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 04:52:05 (1769075525) [ 2526.688368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2526.693964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2526.718817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2527.202804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.227364] Lustre: Skipped 1 previous similar message [ 2527.242090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2527.251481] Lustre: Skipped 2 previous similar messages [ 2529.666395] LustreError: 92750:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2529.915880] Lustre: server umount lustre-MDT0000 complete [ 2532.331667] LustreError: 89815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2532.357641] LustreError: 89815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2533.726130] LustreError: 89799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075541 with bad export cookie 569757188916048399 [ 2533.727436] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2533.736226] LustreError: 89799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2533.948705] LustreError: 92952:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2533.959904] LustreError: 92952:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2534.208602] Lustre: server umount lustre-MDT0001 complete [ 2548.023147] LustreError: 93153:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2548.026211] LustreError: 93153:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2548.285866] Lustre: server umount lustre-OST0000 complete [ 2558.856251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.177836] LustreError: 93783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.355468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2563.225820] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2564.579106] LustreError: 93783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.688511] LustreError: 93783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.106994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2574.028869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2578.931600] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2584.864783] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2588.587877] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2594.019691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2599.803764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2599.879605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2600.043975] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2600.051918] LustreError: 93771:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2600.069143] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2600.078458] LustreError: 95644:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2600.081871] LustreError: 95644:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2600.093972] LustreError: 95644:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2600.154379] Lustre: server umount lustre-OST0000 complete [ 2600.157783] LustreError: 95644:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2608.345447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2608.545600] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2608.552682] LustreError: 93771:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2608.569404] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2608.574452] LustreError: 96110:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2608.585560] LustreError: 96110:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2608.593454] LustreError: 96110:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2608.660546] Lustre: server umount lustre-OST0000 complete [ 2608.665437] LustreError: 96110:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2611.480860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2617.479943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2617.630827] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2617.634334] Lustre: Found index 0 for lustre-OST0000, updating log [ 2617.638241] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2617.710249] Lustre: lustre-OST0000: new disk, initializing [ 2617.715650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2617.971274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2624.069470] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2628.070774] Lustre: Failing over lustre-OST0000 [ 2628.136218] LustreError: 97199:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2628.147408] LustreError: 97199:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2628.276789] Lustre: server umount lustre-OST0000 complete [ 2635.233399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2635.254177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2635.288635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2638.146059] LustreError: 97401:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2638.151184] LustreError: 97401:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2638.275556] Lustre: server umount lustre-MDT0000 complete [ 2640.368661] LustreError: 97402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.406480] LustreError: 97402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2642.482206] LustreError: 93763:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075649 with bad export cookie 569757188916051136 [ 2642.487608] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2642.488477] LustreError: 93763:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2652.132448] LustreError: 97603:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2652.150713] LustreError: 97603:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2652.313596] Lustre: server umount lustre-MDT0001 complete [ 2662.038378] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 04:54:28 (1769075668) [ 2680.640910] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2692.949858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2693.376518] LustreError: 98976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2693.495272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2697.048814] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2705.067144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2710.152678] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2713.257288] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2720.525517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2721.087726] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2727.716478] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2730.275687] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2730.326419] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2730.327399] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2736.484462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2736.563437] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2736.605746] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2736.714385] Lustre: lustre-OST0001: new disk, initializing [ 2736.719372] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2736.981673] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2736.987194] Lustre: Skipped 2 previous similar messages [ 2743.187709] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2745.486445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2745.497456] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2745.648548] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2750.723360] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2757.820261] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2771.117458] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.51@tcp (stopping) [ 2771.122391] Lustre: Skipped 1 previous similar message [ 2771.435868] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2771.455598] Lustre: Skipped 2 previous similar messages [ 2773.778197] LustreError: 102219:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2773.789828] LustreError: 102219:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2773.901729] Lustre: server umount lustre-MDT0001 complete [ 2776.223091] LustreError: 100460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2776.244245] LustreError: 100460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2776.816957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2785.172655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2785.344429] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2785.352400] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2785.463093] Lustre: 98965:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2786.630156] LustreError: 102785:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2786.638499] LustreError: 102785:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2786.644922] LustreError: 98967:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2786.731481] LustreError: 102794:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.203.151@tcp: cfg command failed: rc = -17 [ 2786.735535] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2786.735535] [ 2791.150636] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2791.430853] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2802.083883] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 04:56:48 (1769075808) [ 2820.815387] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2841.433914] Lustre: 104832:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2866.822870] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2870.823613] Lustre: 105966:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2970.083658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2970.092515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2970.109894] Lustre: Skipped 1 previous similar message [ 2970.119970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2970.124737] Lustre: Skipped 3 previous similar messages [ 2975.987405] LustreError: 106421:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2975.998744] LustreError: 106421:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2976.185839] Lustre: server umount lustre-MDT0000 complete [ 2980.189952] LustreError: 101955:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075987 with bad export cookie 569757188916052144 [ 2980.192879] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2980.203487] LustreError: 101955:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2980.843631] LustreError: 98972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2980.849265] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2980.865909] LustreError: 98972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2980.902274] Lustre: Skipped 3 previous similar messages [ 2980.963806] Lustre: server umount lustre-MDT0001 complete [ 2987.972771] Lustre: server umount lustre-OST0000 complete [ 2992.644680] Lustre: server umount lustre-OST0001 complete [ 2999.213958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3010.253412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3011.409401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3020.740307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3031.409606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3032.469840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3055.446276] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 3067.545437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3067.573813] Lustre: lustre-MDT0000: reset Object Index mappings [ 3068.120715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3068.129338] Lustre: Skipped 1 previous similar message [ 3073.347825] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3083.574182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3083.596248] Lustre: lustre-MDT0001: reset Object Index mappings [ 3089.196493] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3093.011238] Lustre: 111035:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3101.507815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3108.851682] LustreError: 111391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3108.873103] LustreError: 111391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3108.893934] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3110.682636] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3112.950699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3120.555368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3123.023455] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3123.051313] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3128.601472] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3137.651046] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3141.641773] Lustre: 112893:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3201.676463] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:03:27 (1769076207) [ 3227.619123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3227.623810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.630876] Lustre: Skipped 1 previous similar message [ 3227.634773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3227.638020] Lustre: Skipped 7 previous similar messages [ 3232.032555] LustreError: 113243:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3232.040770] LustreError: 113243:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3232.225352] Lustre: server umount lustre-MDT0000 complete [ 3236.286737] LustreError: 109898:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769076243 with bad export cookie 569757188916099051 [ 3236.294203] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3236.302129] LustreError: 109898:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3242.900753] Lustre: server umount lustre-MDT0001 complete [ 3253.238511] Lustre: server umount lustre-OST0000 complete [ 3264.782301] Lustre: server umount lustre-OST0001 complete [ 3273.219950] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 3286.600302] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 3298.398785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3307.758541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3315.637935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3324.158312] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3332.308958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3332.387804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3332.625424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3332.675049] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3332.746959] Lustre: lustre-MDT0000: new disk, initializing [ 3332.873203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3332.879921] Lustre: Skipped 3 previous similar messages [ 3332.896052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3337.716078] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3346.979960] Lustre: server umount lustre-MDT0000 complete [ 3350.043489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3357.567581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3357.652423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.939221] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3357.954901] Lustre: 117128:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3357.968415] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3363.305355] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3368.804626] LustreError: 117641:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3368.812947] LustreError: 117641:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 3369.055097] Lustre: server umount lustre-MDT0000 complete [ 3379.208488] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 05:06:25 (1769076385) [ 3380.751635] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3382.391312] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 05:06:29 (1769076389) [ 3401.254903] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 3417.837889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3425.180694] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3436.697505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3436.756226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3436.853583] Lustre: 119145:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3436.931624] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3436.935619] Lustre: Skipped 1 previous similar message [ 3436.992630] Lustre: lustre-MDT0001: new disk, initializing [ 3437.127727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3437.141378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3442.198449] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3447.681223] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3456.025082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3456.133145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3456.591166] Lustre: lustre-OST0000: new disk, initializing [ 3456.600459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3458.210439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3458.226165] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3458.498841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3464.985780] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3478.059354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3478.173917] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3478.431293] Lustre: lustre-OST0001: new disk, initializing [ 3478.435162] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3480.425336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3480.439427] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3480.639234] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3484.821769] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3496.212765] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3500.485371] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3931.572387] Lustre: 127960:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3931.581555] Lustre: 127960:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3931.589479] Lustre: 127960:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3931.599958] Lustre: 127960:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3931.608427] Lustre: 127960:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3931.613915] Lustre: 127960:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4275.658272] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:21:21 (1769077281) [ 4292.064531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4292.074528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4292.083811] Lustre: Skipped 5 previous similar messages [ 4292.093166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4292.098337] Lustre: Skipped 8 previous similar messages [ 4294.634064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4294.649729] Lustre: Skipped 3 previous similar messages [ 4297.755220] LustreError: 128208:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4297.759756] LustreError: 128208:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4297.933714] Lustre: server umount lustre-MDT0000 complete [ 4299.751897] LustreError: 119158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4299.775557] LustreError: 119158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 4302.094924] LustreError: 119140:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769077309 with bad export cookie 569757188916166335 [ 4302.109758] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4302.114222] LustreError: 119140:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4304.866863] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4304.880677] Lustre: Skipped 3 previous similar messages [ 4304.894391] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4304.901204] Lustre: Skipped 1 previous similar message [ 4308.632302] Lustre: server umount lustre-MDT0001 complete [ 4313.883091] Lustre: server umount lustre-OST0000 complete [ 4318.010041] Lustre: server umount lustre-OST0001 complete [ 4325.604376] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 4337.254103] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 4349.933779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4359.637889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4367.250881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4375.759228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4383.858425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4383.945352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4384.204334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4384.253974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4384.335371] Lustre: lustre-MDT0000: new disk, initializing [ 4384.437897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4384.442406] Lustre: Skipped 5 previous similar messages [ 4384.456788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4388.364897] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4401.501445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4401.564517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4401.639646] Lustre: 131002:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4401.701716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4401.705081] Lustre: Skipped 1 previous similar message [ 4401.762834] Lustre: lustre-MDT0001: new disk, initializing [ 4401.882487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4401.906820] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4407.622946] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4416.761135] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4422.147844] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4430.115323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4430.199593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4430.361470] Lustre: lustre-OST0000: new disk, initializing [ 4430.364205] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4430.434730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4430.440962] Lustre: Skipped 1 previous similar message [ 4432.280317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4432.289148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4432.405425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4437.923244] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4446.835299] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4460.563634] Lustre: lustre-MDD0000: changelog on [ 4463.881449] Lustre: lustre-MDD0001: changelog on [ 6434.494266] Lustre: lustre-MDD0001: changelog off [ 6436.095627] Lustre: 131009:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6436.742542] Lustre: lustre-MDD0000: changelog off [ 6444.519537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6444.532011] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6444.548753] Lustre: Skipped 1 previous similar message [ 6444.550602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6444.550609] Lustre: Skipped 2 previous similar messages [ 6444.585534] LustreError: Skipped 1 previous similar message [ 6447.297781] LustreError: 140864:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6447.300504] LustreError: 140864:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 6447.366996] Lustre: server umount lustre-OST0000 complete [ 6450.145479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6450.153463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6450.161973] Lustre: Skipped 1 previous similar message [ 6450.169297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6450.172814] Lustre: Skipped 1 previous similar message [ 6454.753760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6454.755563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6456.133617] LustreError: 141065:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6456.140730] LustreError: 141065:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6456.328697] Lustre: server umount lustre-MDT0000 complete [ 6459.821926] LustreError: 132751:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079467 with bad export cookie 569757188916987225 [ 6459.829876] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6459.833393] LustreError: 132751:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6460.290475] Lustre: server umount lustre-MDT0001 complete [ 6468.845552] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 6470.806183] Key type lgssc unregistered [ 6471.036762] LNet: 141799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6471.047232] LNetError: 141799:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6471.064715] LNet: Removed LNI 192.168.203.151@tcp [ 6471.641137] Key type .llcrypt unregistered [ 6471.642763] Key type ._llcrypt unregistered [ 6488.623816] Key type ._llcrypt registered [ 6488.626116] Key type .llcrypt registered [ 6488.717283] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 6500.780674] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 6501.485085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6501.517084] alg: No test for adler32 (adler32-zlib) [ 6502.592272] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 6502.793511] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 6504.455188] Key type lgssc registered [ 6505.305330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6512.920084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6518.115336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6523.295989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6528.496901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6534.672775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6534.706417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6534.727647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.903952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6535.927590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6535.972625] Lustre: lustre-MDT0000: new disk, initializing [ 6536.041100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6536.054562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6539.268331] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6547.597639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6547.648767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6547.717821] Lustre: 144955:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6547.755666] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6547.757885] Lustre: Skipped 1 previous similar message [ 6547.804339] Lustre: lustre-MDT0001: new disk, initializing [ 6547.892059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6547.926845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6547.937678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6551.276477] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6557.108250] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6559.933984] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6565.657184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6565.751265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6565.998539] Lustre: lustre-OST0000: new disk, initializing [ 6566.001904] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6566.085227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6571.408200] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6573.071815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6573.076503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6573.153628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6577.923601] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6583.379090] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_module obdecho/obdecho [ 6584.809426] LustreError: 147576:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6584.815478] LustreError: 147576:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 6586.065223] LustreError: 147672:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 6598.628315] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6598.631604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6598.641671] Lustre: Skipped 2 previous similar messages [ 6598.650501] Lustre: Skipped 2 previous similar messages [ 6600.331352] LustreError: 148013:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6600.389592] Lustre: server umount lustre-MDT0000 complete [ 6602.941538] LustreError: 144947:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079610 with bad export cookie 3875128517020351792 [ 6602.949480] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6602.951921] LustreError: 144947:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6603.085498] LustreError: 148214:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6603.093943] LustreError: 148214:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6603.213736] Lustre: server umount lustre-MDT0001 complete [ 6616.190706] LustreError: 148414:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6616.197575] LustreError: 148414:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6616.313511] Lustre: server umount lustre-OST0000 complete [ 6630.526995] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 6637.347660] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 6644.311142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6648.840904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6653.560366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6658.421418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6663.054408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6663.087825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6663.225893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6663.246964] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6663.326206] Lustre: lustre-MDT0000: new disk, initializing [ 6663.390184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6663.407300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6665.970905] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6672.536920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6672.579789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6672.625294] Lustre: 151369:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6672.667751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6672.669218] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6672.671026] Lustre: Skipped 1 previous similar message [ 6672.724359] Lustre: lustre-MDT0001: new disk, initializing [ 6672.784637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6672.826640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6672.833383] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6675.190315] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6679.916486] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6682.493068] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6687.018351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6687.085874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6687.319506] Lustre: lustre-OST0000: new disk, initializing [ 6687.323184] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6687.388464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6687.576622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6687.586329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6687.648241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6690.909526] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6696.207935] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6699.432178] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6699.543590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6702.551201] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6702.734271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6708.192874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6708.197109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6708.203488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6708.208767] Lustre: Skipped 1 previous similar message [ 6710.072095] LustreError: 154309:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6710.075500] LustreError: 154309:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6710.175438] Lustre: server umount lustre-OST0000 complete [ 6713.319845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6713.341697] Lustre: Skipped 1 previous similar message [ 6713.344705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6713.349710] Lustre: Skipped 1 previous similar message [ 6718.432384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6718.439479] Lustre: Skipped 1 previous similar message [ 6718.860948] LustreError: 154510:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6718.866110] LustreError: 154510:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6718.938451] Lustre: server umount lustre-MDT0000 complete [ 6721.181278] LustreError: 152152:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079728 with bad export cookie 3875128517020352751 [ 6721.182516] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6721.189511] LustreError: 152152:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6721.384059] Lustre: server umount lustre-MDT0001 complete [ 6723.625980] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:02:10 (1769079730) [ 6724.513852] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6725.496189] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 06:02:12 (1769079732) [ 6731.041987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6731.374651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6734.369659] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6736.864839] LustreError: 155384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6736.881581] LustreError: 155384:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6739.830809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6743.054321] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6746.898418] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6749.493703] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6753.468666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6757.744367] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6762.246368] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6771.169451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6771.176114] LustreError: Skipped 1 previous similar message [ 6771.179995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6771.191399] Lustre: Skipped 1 previous similar message [ 6771.194042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6773.855031] LustreError: 157628:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6773.858735] LustreError: 157628:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6773.911972] Lustre: server umount lustre-MDT0000 complete [ 6775.270819] LustreError: 155384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6775.273347] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6776.144570] LustreError: 155364:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079783 with bad export cookie 3875128517020353633 [ 6776.157882] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6777.994532] LustreError: 157538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6778.002730] LustreError: 157538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6780.387121] LustreError: 155378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6782.398272] Lustre: server umount lustre-MDT0001 complete [ 6786.364276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6794.184985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6794.328784] LustreError: 158890:0:(llog.c:1643:llog_backup()) MGC192.168.203.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6794.334620] Lustre: 158890:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6798.536626] LustreError: 158898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6798.544368] LustreError: 158898:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6809.739180] LustreError: 158897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6809.766291] LustreError: 158897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6826.463828] LustreError: 158896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6826.476322] LustreError: 158896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 6855.647569] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6855.650314] Lustre: 158890:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.151@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6859.913753] LustreError: 158916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6859.942443] LustreError: 158916:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [ 6883.807309] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6883.817939] Lustre: 155375:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.151@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6896.150094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6896.154506] Lustre: Skipped 2 previous similar messages [ 6899.186619] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6901.230986] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6904.292149] LustreError: 155375:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6906.726169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6920.675189] LustreError: 143164:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009a66c710 x1855014316392704/t0(0) o250->MGC192.168.203.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6923.228180] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6926.319324] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6926.904777] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6927.929708] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6930.193715] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6931.475769] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6946.783949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6946.786136] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6946.788104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6946.788112] Lustre: Skipped 4 previous similar messages [ 6946.811739] Lustre: Skipped 4 previous similar messages [ 6949.795074] LustreError: 160395:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6949.797237] LustreError: 160395:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6949.890395] Lustre: server umount lustre-MDT0000 complete [ 6951.908461] LustreError: 160286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6951.922717] LustreError: 160286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 46 previous similar messages [ 6952.313907] LustreError: 155365:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079959 with bad export cookie 3875128517020360143 [ 6952.315102] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6952.320625] LustreError: 155365:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6952.540272] Lustre: server umount lustre-MDT0001 complete [ 6965.223870] Lustre: server umount lustre-OST0000 complete [ 6978.408245] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 6984.404838] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 6990.816401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6995.017664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6998.820569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7002.764992] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7006.625216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7006.662796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7006.818154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7006.834871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7006.884204] Lustre: lustre-MDT0000: new disk, initializing [ 7006.954427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7006.960987] Lustre: Skipped 1 previous similar message [ 7006.971339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7009.006577] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7015.413616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7015.437109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7015.476947] Lustre: 163741:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7015.596127] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7017.664079] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7022.611704] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7024.701834] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7028.793662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7028.843455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7032.463468] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7035.929159] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7035.957060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7036.975507] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7039.375416] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7039.499776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7041.657238] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7041.791173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7046.116630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7046.117828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7046.125202] LustreError: Skipped 1 previous similar message [ 7046.133789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7046.142839] Lustre: Skipped 3 previous similar messages [ 7057.376338] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7057.410098] LustreError: 166679:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7057.412854] LustreError: 166679:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7057.476614] Lustre: server umount lustre-OST0000 complete [ 7065.772767] Lustre: server umount lustre-MDT0000 complete [ 7067.689405] LustreError: 163734:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080075 with bad export cookie 3875128517020376929 [ 7067.693904] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7067.693918] LustreError: 163734:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7067.866910] Lustre: server umount lustre-MDT0001 complete [ 7071.951128] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 06:07:59 (1769080079) [ 7075.063878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7077.104097] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7080.417904] LustreError: 167757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7080.423875] LustreError: 167757:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7080.881144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7082.862415] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7085.594790] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7087.985901] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7091.759954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7095.069744] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7098.232150] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7112.747068] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 06:08:40 (1769080120) [ 7115.744123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7115.751163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7115.761758] Lustre: Skipped 3 previous similar messages [ 7115.767102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7115.773246] Lustre: Skipped 6 previous similar messages [ 7120.853710] Lustre: server umount lustre-OST0000 complete [ 7127.008854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7127.012281] LustreError: Skipped 1 previous similar message [ 7129.793317] LustreError: 170420:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7129.796130] LustreError: 170420:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 7129.832628] Lustre: server umount lustre-MDT0000 complete [ 7131.418705] LustreError: 167737:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080138 with bad export cookie 3875128517020377804 [ 7131.420396] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7131.424193] LustreError: 167737:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7131.541978] Lustre: server umount lustre-MDT0001 complete [ 7135.721572] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 7136.935149] Key type lgssc unregistered [ 7137.086580] LNet: 171153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7137.092533] LNetError: 171153:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7137.101355] LNet: Removed LNI 192.168.203.151@tcp [ 7137.438576] Key type .llcrypt unregistered [ 7137.439784] Key type ._llcrypt unregistered [ 7145.742618] Key type ._llcrypt registered [ 7145.744056] Key type .llcrypt registered [ 7145.791776] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7146.115646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7146.126723] alg: No test for adler32 (adler32-zlib) [ 7147.011632] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 7147.129455] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 7148.727837] Key type lgssc registered [ 7149.245043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7152.995662] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7153.010469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7168.479399] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7173.599333] LustreError: 172353:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.151@tcp: failed processing log, type 4: rc = -110 [ 7178.719467] LustreError: 172353:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7178.737126] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7178.743325] LustreError: MGC192.168.203.151@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7178.753052] LustreError: 172353:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7178.759297] LustreError: 172353:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 7178.764590] LustreError: 172353:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 7178.768837] LustreError: 172353:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 7178.850356] Lustre: server umount lustre-OST0000 complete [ 7178.857669] LustreError: 172353:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7184.933327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7186.297881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7189.007548] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7195.154573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7195.328827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7197.802421] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7201.880339] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7203.814553] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7207.919268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7208.100225] LustreError: 174445:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7208.107709] Lustre: 174445:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.203.151@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7208.208649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7211.712066] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7215.003190] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7216.608530] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7216.612282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7216.623978] Lustre: Skipped 1 previous similar message [ 7216.635113] Lustre: Skipped 1 previous similar message [ 7220.706847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7220.710845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7220.722809] Lustre: Skipped 2 previous similar messages [ 7222.287638] LustreError: 175097:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7222.290246] LustreError: 175097:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7222.333086] Lustre: server umount lustre-MDT0000 complete [ 7224.318961] LustreError: 174473:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080231 with bad export cookie 4163196415750181171 [ 7224.319852] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7224.324223] LustreError: 174473:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7224.823378] LustreError: 173567:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000077b84731 x1855014991583616/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7224.849255] LustreError: 175300:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7224.851706] LustreError: 175300:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7224.946389] Lustre: server umount lustre-MDT0001 complete [ 7237.559525] LustreError: 175501:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7237.561882] LustreError: 175501:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7237.630570] Lustre: server umount lustre-OST0000 complete [ 7242.323852] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 7247.421207] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7252.799076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7256.716258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7260.029429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7263.732274] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7267.290820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7267.321393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7267.431045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7267.451956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7267.502651] Lustre: lustre-MDT0000: new disk, initializing [ 7267.546210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7267.560292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7269.547627] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7275.185228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7275.210688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7275.256575] Lustre: 177836:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7275.294804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7275.297803] Lustre: Skipped 1 previous similar message [ 7275.336381] Lustre: lustre-MDT0001: new disk, initializing [ 7275.379465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7275.404573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7275.412153] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7277.270716] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7280.994428] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7282.734612] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7286.140215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7286.188181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7286.317412] Lustre: lustre-OST0000: new disk, initializing [ 7286.321375] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7286.380654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7287.571813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7287.576966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7287.632404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7289.287802] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7293.628836] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7295.642316] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7295.760995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7297.429309] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7297.527799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7303.137502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7303.137528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7303.139537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7303.142544] Lustre: Skipped 1 previous similar message [ 7303.150341] LustreError: Skipped 1 previous similar message [ 7304.431101] LustreError: 180776:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7304.434089] LustreError: 180776:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7304.486281] Lustre: server umount lustre-OST0000 complete [ 7306.208076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7306.211256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7306.215609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7306.222756] Lustre: Skipped 1 previous similar message [ 7312.314953] Lustre: server umount lustre-MDT0000 complete [ 7313.376243] LustreError: 180777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7313.384929] LustreError: 180777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7314.327435] LustreError: 177827:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080321 with bad export cookie 4163196415750181892 [ 7314.330363] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7314.332128] LustreError: 177827:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7314.424878] LustreError: 181180:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7314.427186] LustreError: 181180:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 7314.513302] Lustre: server umount lustre-MDT0001 complete [ 7319.396909] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 06:12:06 (1769080326) [ 7328.983571] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 7330.458146] Key type lgssc unregistered [ 7330.625394] LNet: 182362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7330.629278] LNetError: 182362:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7330.638567] LNet: Removed LNI 192.168.203.151@tcp [ 7331.041532] Key type .llcrypt unregistered [ 7331.042985] Key type ._llcrypt unregistered [ 7341.706719] Key type ._llcrypt registered [ 7341.708661] Key type .llcrypt registered [ 7341.779836] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7342.180713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7342.189768] alg: No test for adler32 (adler32-zlib) [ 7343.077901] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 7343.198652] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 7344.800597] Key type lgssc registered [ 7345.397623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7349.352771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7349.366173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7350.611397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7352.794184] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7357.279522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7357.447577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7359.212784] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7361.512956] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7363.103363] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7366.602760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7366.695925] LustreError: 185183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7366.705668] LustreError: 185183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7366.715991] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7366.720822] LustreError: 185175:0:(llog.c:1684:llog_backup()) MGC192.168.203.151@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7366.726111] Lustre: 185175:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.203.151@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7366.794504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7369.661472] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7372.051273] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7377.378079] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7377.381614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7377.383709] Lustre: Skipped 1 previous similar message [ 7379.265801] LustreError: 185829:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7379.295195] Lustre: server umount lustre-MDT0000 complete [ 7380.854318] LustreError: 183517:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080388 with bad export cookie 13946012511622781074 [ 7380.857342] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7380.859604] LustreError: 183517:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7380.930226] LustreError: 186030:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7380.933125] LustreError: 186030:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7380.996171] Lustre: server umount lustre-MDT0001 complete [ 7392.759966] LustreError: 186234:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7392.763200] LustreError: 186234:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7392.819676] Lustre: server umount lustre-OST0000 complete [ 7396.880930] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 7401.575483] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7406.377263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7409.760810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7412.686876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7415.783505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7419.176658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7419.218059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7419.340565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7419.363278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7419.407252] Lustre: lustre-MDT0000: new disk, initializing [ 7419.449138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7419.462095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7421.177747] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7427.169589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7427.197969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7427.247330] Lustre: 188569:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7427.283252] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7427.285606] Lustre: Skipped 1 previous similar message [ 7427.287807] LustreError: 189336:0:(llog.c:1684:llog_backup()) MGC192.168.203.151@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7427.291897] LustreError: 189336:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 7427.295765] Lustre: 189336:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.151@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7427.306923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7427.309713] Lustre: Skipped 1 previous similar message [ 7427.349168] Lustre: lustre-MDT0001: new disk, initializing [ 7427.403632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7427.428951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7427.434213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7429.334511] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7433.683784] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7435.457225] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7438.762925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7438.805822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7438.925962] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7438.929436] Lustre: Skipped 1 previous similar message [ 7438.931658] LustreError: 190291:0:(llog.c:1684:llog_backup()) MGC192.168.203.151@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7438.936410] LustreError: 190291:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 7438.941122] Lustre: 190291:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.151@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7438.946349] Lustre: 190291:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7438.961230] Lustre: lustre-OST0000: new disk, initializing [ 7438.964118] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7439.005547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7441.752969] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7445.768518] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7447.048134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7447.056843] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7447.121120] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7447.792932] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7447.878755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7449.797604] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7449.907146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7452.130340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7452.131991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7452.132258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7452.132263] Lustre: Skipped 2 previous similar messages [ 7452.135713] Lustre: Skipped 2 previous similar messages [ 7456.942202] LustreError: 191507:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7456.946653] LustreError: 191507:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7457.003474] Lustre: server umount lustre-OST0000 complete [ 7462.368683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7462.370190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7462.376166] Lustre: Skipped 1 previous similar message [ 7462.381193] Lustre: Skipped 2 previous similar messages [ 7464.775916] LustreError: 191708:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7464.778102] LustreError: 191708:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7464.823364] Lustre: server umount lustre-MDT0000 complete [ 7466.706357] LustreError: 188561:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080474 with bad export cookie 13946012511622781774 [ 7466.707369] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7466.715904] LustreError: 188561:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7466.860666] Lustre: server umount lustre-MDT0001 complete [ 7470.886776] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 06:14:38 (1769080478) [ 7478.201274] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7482.644201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7482.852433] LustreError: 193275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7482.876498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7484.780310] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7487.968641] LustreError: 193275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7488.824585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7490.664747] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7491.998803] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7494.738603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7497.417437] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7501.289398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7501.320929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7501.395543] Lustre: lustre-OST0001: new disk, initializing [ 7501.398417] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7503.754668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7503.760939] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7503.807214] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7503.854952] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7510.089684] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7515.438386] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7521.357698] Lustre: DEBUG MARKER: ADD OST3 [ 7523.528662] loop: module loaded [ 7523.554884] loop0: detected capacity change from 0 to 204800000 [ 7523.686286] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7523.691499] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7523.698865] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7523.759432] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7526.381936] loop0: detected capacity change from 0 to 204800000 [ 7528.708850] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7528.739625] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7528.812654] Lustre: lustre-OST0003: new disk, initializing [ 7528.815395] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7528.859867] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7528.866156] Lustre: Skipped 3 previous similar messages [ 7529.027941] Lustre: DEBUG MARKER: STOP OST3 [ 7529.852689] Lustre: Failing over lustre-OST0003 [ 7529.874236] LustreError: 197524:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 7529.880962] LustreError: 197524:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7529.907991] Lustre: server umount lustre-OST0003 complete [ 7529.948679] LustreError: 195871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7531.436901] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7535.079100] LustreError: 195871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7535.088710] LustreError: 195871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7535.869095] Lustre: DEBUG MARKER: START OST3 again [ 7539.294257] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7540.989202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7540.996711] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7541.011073] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 7542.257190] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7558.145377] Lustre: Failing over lustre-OST0003 [ 7558.204783] LustreError: 199564:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 7558.207836] LustreError: 199564:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7558.232654] Lustre: server umount lustre-OST0003 complete [ 7560.166359] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7560.176356] LustreError: 195871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7560.184080] LustreError: 195871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7575.522126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7575.525092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7575.527537] Lustre: Skipped 3 previous similar messages [ 7575.532073] Lustre: Skipped 3 previous similar messages [ 7577.717309] Lustre: server umount lustre-MDT0000 complete [ 7579.262057] LustreError: 193254:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080586 with bad export cookie 13946012511622782642 [ 7579.265846] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7579.266799] LustreError: 193254:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7580.640256] LustreError: 194748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7580.642083] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7580.649879] LustreError: 194748:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 7580.655954] Lustre: Skipped 1 previous similar message [ 7585.469371] Lustre: server umount lustre-MDT0001 complete [ 7586.997950] Lustre: server umount lustre-OST0000 complete [ 7588.503919] Lustre: server umount lustre-OST0001 complete [ 7591.118504] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 7594.493898] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7598.585524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7601.353201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7604.006706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7606.821253] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7609.754263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7609.777438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7609.869130] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7609.884328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7609.923844] Lustre: lustre-MDT0000: new disk, initializing [ 7609.970829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7609.975267] Lustre: Skipped 1 previous similar message [ 7609.985225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7611.699239] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7617.028815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7617.059260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7617.100618] Lustre: 202618:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7617.235380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7619.141822] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7622.926351] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7624.574737] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7627.648970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7627.680154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7627.811268] Lustre: lustre-OST0000: new disk, initializing [ 7627.812966] Lustre: Skipped 1 previous similar message [ 7627.815039] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7627.817427] Lustre: Skipped 2 previous similar messages [ 7629.341202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7629.344046] Lustre: Skipped 1 previous similar message [ 7629.345865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7629.398335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7630.348897] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7634.284372] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7636.246743] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7636.343037] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7638.118575] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7638.215626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7639.524612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7639.525632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7639.525811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7639.525816] LustreError: Skipped 1 previous similar message [ 7639.533056] Lustre: Skipped 4 previous similar messages [ 7639.535420] Lustre: Skipped 2 previous similar messages [ 7645.230212] LustreError: 205557:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7645.232931] LustreError: 205557:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 7645.282015] Lustre: server umount lustre-OST0000 complete [ 7648.224016] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7648.227086] LustreError: Skipped 1 previous similar message [ 7652.967349] Lustre: server umount lustre-MDT0000 complete [ 7654.523925] LustreError: 202609:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080661 with bad export cookie 13946012511622784903 [ 7654.524787] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7654.528789] LustreError: 202609:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7654.664745] Lustre: server umount lustre-MDT0001 complete [ 7656.224978] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 06:17:43 (1769080663) [ 7663.379600] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 7667.999723] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7672.475680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7675.512386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7678.463563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7681.281238] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7684.228990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7684.262773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7684.357229] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7684.372000] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7684.410541] Lustre: lustre-MDT0000: new disk, initializing [ 7684.465327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7686.179995] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7691.143859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7691.171298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7691.212300] Lustre: 208804:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7691.337142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7692.985381] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7696.604509] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7698.205067] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7701.102162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7701.134399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7702.833457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7703.617288] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7707.313217] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7709.234551] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7709.331159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7711.050472] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7711.158232] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7713.251631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7713.254235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7713.255583] LustreError: Skipped 1 previous similar message [ 7713.259797] Lustre: Skipped 2 previous similar messages [ 7713.263425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7713.266384] Lustre: Skipped 7 previous similar messages [ 7718.126262] LustreError: 211743:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7718.132317] LustreError: 211743:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7718.192546] Lustre: server umount lustre-OST0000 complete [ 7721.952220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7734.239132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7734.317374] Lustre: server umount lustre-MDT0000 complete [ 7736.017213] LustreError: 211946:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080743 with bad export cookie 13946012511622785785 [ 7736.020608] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7736.022884] LustreError: 211946:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7736.147229] Lustre: server umount lustre-MDT0001 complete [ 7740.339112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7740.529750] LustreError: 212625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7740.535388] LustreError: 212625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7740.563182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7740.565332] Lustre: Skipped 5 previous similar messages [ 7743.134081] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7746.875845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7748.914114] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7751.734334] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7753.591507] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7756.962722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7759.528840] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7761.921523] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7776.738459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7776.739169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7776.741845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7776.745907] Lustre: Skipped 3 previous similar messages [ 7776.748635] Lustre: Skipped 7 previous similar messages [ 7780.482186] Lustre: server umount lustre-OST0000 complete [ 7788.576348] Lustre: server umount lustre-MDT0000 complete [ 7790.346737] LustreError: 212606:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080797 with bad export cookie 13946012511622786660 [ 7790.350453] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7790.354510] LustreError: 212606:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7790.524266] Lustre: server umount lustre-MDT0001 complete [ 7896.624892] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 7897.975462] Key type lgssc unregistered [ 7898.136040] LNet: 215956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7898.139646] LNetError: 215956:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7898.155503] LNet: Removed LNI 192.168.203.151@tcp [ 7898.564166] Key type .llcrypt unregistered [ 7898.565965] Key type ._llcrypt unregistered [ 7909.275425] Key type ._llcrypt registered [ 7909.276609] Key type .llcrypt registered [ 7909.327618] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 7916.606713] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7917.042114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7917.055646] alg: No test for adler32 (adler32-zlib) [ 7917.931950] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 7918.046108] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 7919.655183] Key type lgssc registered [ 7920.108163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7923.871878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7926.548720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7928.858632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7931.661570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7934.497696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7934.530400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7934.538734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7935.639482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7935.655154] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7935.696583] Lustre: lustre-MDT0000: new disk, initializing [ 7935.750474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7935.763866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7937.405870] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7942.169809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7942.197612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7942.234071] Lustre: 219256:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7942.260536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7942.262582] Lustre: Skipped 1 previous similar message [ 7942.308428] Lustre: lustre-MDT0001: new disk, initializing [ 7942.354151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7942.370105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7942.373923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7943.842130] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7947.331439] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7948.785147] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7951.238440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7951.263064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7951.350363] Lustre: lustre-OST0000: new disk, initializing [ 7951.352451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7951.380883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7953.360220] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7956.515423] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7957.973197] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7959.032772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7959.035748] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7959.068435] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7960.089834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 7961.480849] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7961.551595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7964.130051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7964.132410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7964.133612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7964.137608] Lustre: Skipped 1 previous similar message [ 7964.141648] Lustre: Skipped 1 previous similar message [ 7968.370131] LustreError: 222211:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7968.416263] Lustre: server umount lustre-OST0000 complete [ 7973.343930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7973.347531] LustreError: Skipped 1 previous similar message [ 7973.349703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7973.355931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7974.368337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7974.368887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7979.488658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7979.491224] Lustre: Skipped 2 previous similar messages [ 7983.584276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7983.587274] Lustre: Skipped 1 previous similar message [ 7984.095117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7984.130610] LustreError: 222414:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7984.133401] LustreError: 222414:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7984.163761] Lustre: server umount lustre-MDT0000 complete [ 7985.466276] LustreError: 221003:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080992 with bad export cookie 14148416370938982656 [ 7985.468252] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7985.471219] LustreError: 221003:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7985.539583] LustreError: 222618:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7985.542009] LustreError: 222618:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7985.598251] Lustre: server umount lustre-MDT0001 complete [ 7992.410173] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7996.905372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7997.111608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7998.562556] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8002.529115] LustreError: 223808:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8002.535943] LustreError: 223808:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8002.594636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8002.753839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8004.365190] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8005.503105] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8007.962798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8010.242806] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8013.424556] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8013.452699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8013.514166] Lustre: lustre-OST0001: new disk, initializing [ 8013.517269] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8013.550398] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8013.554727] Lustre: Skipped 1 previous similar message [ 8015.754775] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8018.943589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8018.947644] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8018.981819] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8025.803592] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8032.885678] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8042.550976] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 06:24:10 (1769081050) [ 8044.001091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8044.004515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8044.015557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8049.758094] LustreError: 227170:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8049.767929] LustreError: 227170:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8049.974144] Lustre: server umount lustre-MDT0000 complete [ 8052.720600] LustreError: 225263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8052.761975] LustreError: 225263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8054.477077] LustreError: 223772:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081061 with bad export cookie 14148416370938983524 [ 8054.487192] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8054.506112] LustreError: 223772:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8054.600432] LustreError: 227371:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 8054.604498] LustreError: 227371:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8054.700640] Lustre: server umount lustre-MDT0001 complete [ 8069.456643] LustreError: 227572:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8069.464314] LustreError: 227572:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8069.500555] Lustre: server umount lustre-OST0000 complete [ 8083.042256] Lustre: server umount lustre-OST0001 complete [ 8090.338726] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 8105.405292] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8117.766673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8123.849649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8128.917413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8134.170957] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8139.417766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8139.462939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8139.629366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8139.652675] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8139.703912] Lustre: lustre-MDT0000: new disk, initializing [ 8139.751978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8139.771430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8143.201235] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8151.312816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8151.384967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8151.477649] Lustre: 229985:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8151.539109] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8151.544567] Lustre: Skipped 1 previous similar message [ 8151.642345] Lustre: lustre-MDT0001: new disk, initializing [ 8151.779519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8151.786794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8154.867272] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8161.041993] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8164.222737] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8169.621847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8169.677855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8169.858720] Lustre: lustre-OST0000: new disk, initializing [ 8169.863207] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8171.715109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8171.723122] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8171.788114] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8174.306854] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8180.817617] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8184.721266] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8184.982727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8188.664442] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8188.835857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8192.482188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8192.487960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8192.489782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8192.489794] Lustre: Skipped 4 previous similar messages [ 8192.551464] Lustre: Skipped 4 previous similar messages [ 8196.793022] LustreError: 232923:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8196.796052] LustreError: 232923:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 8196.909242] Lustre: server umount lustre-OST0000 complete [ 8202.721790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8202.729859] Lustre: Skipped 1 previous similar message [ 8206.033495] Lustre: server umount lustre-MDT0000 complete [ 8207.840811] LustreError: 230671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8207.852850] LustreError: 230671:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8209.064712] LustreError: 229976:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081216 with bad export cookie 14148416370938985638 [ 8209.067293] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8209.073930] LustreError: 229976:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8209.301303] Lustre: server umount lustre-MDT0001 complete [ 8216.362906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8216.658172] LustreError: 233802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8216.709941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8216.716584] Lustre: Skipped 2 previous similar messages [ 8220.037254] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8222.181384] LustreError: 233802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8227.368496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8231.212989] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8235.629179] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8238.719489] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8243.846806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8249.037754] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8253.855831] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8400.864609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8400.874236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8400.879099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8400.914717] Lustre: Skipped 1 previous similar message [ 8400.920822] Lustre: Skipped 4 previous similar messages [ 8404.110119] LustreError: 236086:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8404.113525] LustreError: 236086:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8404.246776] Lustre: server umount lustre-OST0000 complete [ 8413.502494] Lustre: server umount lustre-MDT0000 complete [ 8416.225435] LustreError: 235010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8416.255749] LustreError: 235010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8416.849978] LustreError: 233782:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081424 with bad export cookie 14148416370938986520 [ 8416.855415] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8416.861977] LustreError: 233782:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8417.163869] Lustre: server umount lustre-MDT0001 complete [ 8520.001416] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 8521.919135] Key type lgssc unregistered [ 8522.157466] LNet: 237056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8522.163810] LNetError: 237056:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8522.174069] LNet: Removed LNI 192.168.203.151@tcp [ 8522.688176] Key type .llcrypt unregistered [ 8522.690775] Key type ._llcrypt unregistered [ 8541.603847] Key type ._llcrypt registered [ 8541.606189] Key type .llcrypt registered [ 8541.703772] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 8552.682447] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8553.620994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8553.643299] alg: No test for adler32 (adler32-zlib) [ 8554.605268] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 8554.740075] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 8556.359161] Key type lgssc registered [ 8557.069349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8562.812363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8567.293074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8571.148747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8575.650822] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8585.415704] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8590.992462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8591.035915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8591.047592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8592.172219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8592.188360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8592.237845] Lustre: lustre-MDT0000: new disk, initializing [ 8592.284608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8592.302948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8594.572037] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8600.616693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8600.676417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8600.729386] Lustre: 241059:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8600.758378] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8600.760278] Lustre: Skipped 1 previous similar message [ 8600.799384] Lustre: lustre-MDT0001: new disk, initializing [ 8600.841959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8600.859825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8600.870953] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8602.513463] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8605.394643] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8608.768821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8608.801905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8608.942582] Lustre: lustre-OST0000: new disk, initializing [ 8608.945357] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8609.004384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8611.785152] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8615.942694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8615.947287] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8615.992389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8616.794151] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8616.821173] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8616.890755] Lustre: lustre-OST0001: new disk, initializing [ 8616.892748] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8616.929328] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8619.107686] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8622.102190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8622.106309] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8622.159308] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8625.464974] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8629.591842] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8640.277553] LustreError: 244585:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8640.329047] Lustre: server umount lustre-MDT0000 complete [ 8641.832913] LustreError: 241051:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081649 with bad export cookie 11371028732958014216 [ 8641.836829] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8641.836991] LustreError: 241051:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8642.528798] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8642.530480] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8642.534222] Lustre: Skipped 3 previous similar messages [ 8642.538067] Lustre: Skipped 1 previous similar message [ 8643.968300] LustreError: 244786:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 8643.974274] LustreError: 244786:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8644.030199] Lustre: server umount lustre-MDT0001 complete [ 8651.440130] LustreError: 244988:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8651.443314] LustreError: 244988:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8651.468107] Lustre: server umount lustre-OST0000 complete [ 8652.907472] Lustre: server umount lustre-OST0001 complete [ 8658.727585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8662.128706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8662.535109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8665.859107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8668.992864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8669.386945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8676.967632] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8680.743197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8680.748765] Lustre: lustre-MDT0000: reset Object Index mappings [ 8680.914442] LustreError: 248495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8680.919977] LustreError: 248495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8680.939052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8682.367230] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8685.450381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8685.455440] Lustre: lustre-MDT0001: reset Object Index mappings [ 8687.064955] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8688.119192] Lustre: 249607:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8690.702353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8690.848924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8690.851800] Lustre: Skipped 1 previous similar message [ 8691.876162] LustreError: 250132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8691.882528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8692.970514] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8695.972541] LustreError: 249961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8695.982227] LustreError: 249961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8696.400658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8697.980096] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8699.156448] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8702.817874] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8704.200075] Lustre: 251452:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8711.135787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8711.138148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8711.142758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8712.162839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8712.165316] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8712.171557] Lustre: Skipped 2 previous similar messages [ 8716.678278] LustreError: 251557:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8716.680166] LustreError: 251557:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 8716.713261] Lustre: server umount lustre-MDT0000 complete [ 8717.282206] LustreError: 248489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8717.293398] LustreError: 248489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8718.818133] LustreError: 248476:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081726 with bad export cookie 11371028732958021006 [ 8718.822366] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8718.824166] LustreError: 248476:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8718.977148] Lustre: server umount lustre-MDT0001 complete [ 8730.613907] LustreError: 251961:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 8730.616213] LustreError: 251961:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 8730.632433] Lustre: server umount lustre-OST0000 complete [ 8742.136566] LustreError: 252163:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8742.138669] LustreError: 252163:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8742.194307] Lustre: server umount lustre-OST0001 complete [ 8750.479129] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 06:35:57 (1769081757) [ 8757.465852] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 8761.345304] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8765.266512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8767.982060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8770.646055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8773.186645] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8776.614186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8776.667802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8776.770412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8776.784951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8776.821950] Lustre: lustre-MDT0000: new disk, initializing [ 8776.852037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8776.858419] Lustre: Skipped 1 previous similar message [ 8776.864673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8778.659733] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8784.050262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8784.077673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8784.121152] Lustre: 255680:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8784.253735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8785.890246] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8789.831335] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8791.448234] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8794.314534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8794.349941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8794.484593] Lustre: lustre-OST0000: new disk, initializing [ 8794.487393] Lustre: Skipped 1 previous similar message [ 8794.491659] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8794.493706] Lustre: Skipped 2 previous similar messages [ 8796.033032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8796.036079] Lustre: Skipped 1 previous similar message [ 8796.038407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8796.075115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8796.773284] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8800.399256] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8802.033535] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8802.115360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8803.591853] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8803.675117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8806.368603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8806.368755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8806.371805] LustreError: Skipped 1 previous similar message [ 8806.372499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8806.376784] Lustre: Skipped 1 previous similar message [ 8806.383593] Lustre: Skipped 4 previous similar messages [ 8810.478148] LustreError: 258620:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8810.480638] LustreError: 258620:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8810.523264] Lustre: server umount lustre-OST0000 complete [ 8815.072638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8815.076657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8815.083314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8825.824192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8825.827581] Lustre: Skipped 5 previous similar messages [ 8826.335111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8826.396536] Lustre: server umount lustre-MDT0000 complete [ 8827.704338] LustreError: 256460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081835 with bad export cookie 11371028732958023099 [ 8827.708922] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8827.709049] LustreError: 256460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8827.828683] Lustre: server umount lustre-MDT0001 complete [ 8834.417805] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8838.067034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8838.214860] LustreError: 260198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8838.231842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8838.234139] Lustre: Skipped 2 previous similar messages [ 8839.506595] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8842.612253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8844.241814] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8845.339139] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8847.743889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8849.806335] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8852.844397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8852.870615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8852.926967] Lustre: lustre-OST0001: new disk, initializing [ 8852.929521] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8854.083699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8854.087517] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8854.127711] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8854.958102] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8859.676319] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8860.913112] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8867.904486] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8867.923824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8873.440292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8873.443727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8873.448025] Lustre: Skipped 1 previous similar message [ 8873.450712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8876.097901] LustreError: 263702:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8876.100716] LustreError: 263702:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8876.131238] Lustre: server umount lustre-MDT0000 complete [ 8877.491419] LustreError: 260178:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081884 with bad export cookie 11371028732958023981 [ 8877.495542] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8877.496200] LustreError: 260178:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8877.622321] Lustre: server umount lustre-MDT0001 complete [ 8889.347720] Lustre: server umount lustre-OST0000 complete [ 8901.020721] Lustre: server umount lustre-OST0001 complete [ 8906.479784] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8910.193630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8910.382020] LustreError: 265479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8910.406305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8910.409589] Lustre: Skipped 3 previous similar messages [ 8911.829656] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8914.893183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8916.406798] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8917.456573] Lustre: 266589:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8919.832065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8921.001028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 8921.936667] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8924.951911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8926.129965] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 8927.136871] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8930.554546] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8931.890939] Lustre: 268441:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8938.822731] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8938.869825] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8942.490135] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 06:39:10 (1769081950) [ 8955.872041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8955.874873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8955.879772] Lustre: Skipped 3 previous similar messages [ 8955.882829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8955.885358] Lustre: Skipped 4 previous similar messages [ 8960.127835] LustreError: 269148:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8960.131232] LustreError: 269148:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 8960.164157] Lustre: server umount lustre-MDT0000 complete [ 8961.369172] LustreError: 265459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081968 with bad export cookie 11371028732958030274 [ 8961.372057] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8961.373160] LustreError: 265459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8962.016022] LustreError: 266965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8962.023793] LustreError: 266965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8967.537666] Lustre: server umount lustre-MDT0001 complete [ 8974.912438] Lustre: server umount lustre-OST0000 complete [ 8982.300872] Lustre: server umount lustre-OST0001 complete [ 8984.510499] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 8988.160352] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8991.991456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8994.354265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8996.637288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8998.902749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9004.995784] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9008.851615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9008.873797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9008.957843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9008.973139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9009.011970] Lustre: lustre-MDT0000: new disk, initializing [ 9009.048377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9010.507497] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9015.084520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9015.112942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9015.152382] Lustre: 272631:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9015.255182] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9016.743925] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9021.920944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9021.950643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9022.756106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9024.177665] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9028.530961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9028.559029] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9029.927637] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9030.692330] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9035.261360] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9046.247374] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 06:40:53 (1769082053) [ 9050.594900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9050.596304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9050.600475] Lustre: Skipped 7 previous similar messages [ 9050.603229] Lustre: Skipped 9 previous similar messages [ 9053.222723] Lustre: server umount lustre-MDT0000 complete [ 9054.501152] LustreError: 272622:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082061 with bad export cookie 11371028732958038149 [ 9054.504729] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9054.505954] LustreError: 272622:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9054.613606] Lustre: server umount lustre-MDT0001 complete [ 9066.114059] Lustre: server umount lustre-OST0000 complete [ 9077.603444] Lustre: server umount lustre-OST0001 complete [ 9084.310581] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 9087.962557] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9091.522940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9094.066943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9096.372765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9098.609512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9101.457901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9101.478494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9101.574099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9101.577502] Lustre: Skipped 2 previous similar messages [ 9101.587278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9101.589620] Lustre: Skipped 4 previous similar messages [ 9101.627216] Lustre: lustre-MDT0000: new disk, initializing [ 9101.629100] Lustre: Skipped 3 previous similar messages [ 9101.665782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9101.668723] Lustre: Skipped 7 previous similar messages [ 9101.678748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9101.682463] Lustre: Skipped 3 previous similar messages [ 9103.059960] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9107.318822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9107.343560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9107.380538] Lustre: 279689:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9107.481924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9107.485533] Lustre: Skipped 2 previous similar messages [ 9108.798068] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9112.108350] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9113.412585] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9115.860097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9115.882908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9117.086037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9117.966171] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9121.050928] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9121.822742] Lustre: Mounted lustre-client [ 9122.132932] LustreError: 282190:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff93cfe658c800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 9122.137587] LustreError: 282190:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 9122.140324] LustreError: 282190:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 9122.159125] Lustre: Unmounted lustre-client [ 9126.882603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9126.886285] Lustre: Skipped 2 previous similar messages [ 9127.053965] Lustre: server umount lustre-OST0000 complete [ 9134.424212] Lustre: server umount lustre-MDT0000 complete [ 9135.720411] LustreError: 280469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082143 with bad export cookie 11371028732958039675 [ 9135.722693] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9135.725846] LustreError: 280469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9135.860733] Lustre: server umount lustre-MDT0001 complete [ 9139.599682] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9140.625740] Key type lgssc unregistered [ 9172.255214] LNet: 1 local NIs in recovery (showing 1): 192.168.203.151@tcp [ 9172.511306] LNet: 283467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9172.514024] LNetError: 283467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9254.440075] LNet: Removed LNI 192.168.203.151@tcp [ 9254.783146] Key type .llcrypt unregistered [ 9254.785190] Key type ._llcrypt unregistered [ 9261.754049] Key type ._llcrypt registered [ 9261.755501] Key type .llcrypt registered [ 9261.800559] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 9268.559137] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9268.867811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9268.880052] alg: No test for adler32 (adler32-zlib) [ 9269.751341] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 9269.831447] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 9271.407154] Key type lgssc registered [ 9271.786848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9274.744251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9277.097962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9279.288556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9281.588751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9282.225410] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 06:44:49 (1769082289) [ 9284.458446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9284.484661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9284.492297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9285.584162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9285.595936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9285.629322] Lustre: lustre-MDT0000: new disk, initializing [ 9285.649635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9285.656549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9287.011313] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9291.391678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9291.416641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9291.456152] Lustre: 286636:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9291.483079] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9291.485805] Lustre: Skipped 1 previous similar message [ 9291.525685] Lustre: lustre-MDT0001: new disk, initializing [ 9291.559565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9291.573238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9291.576433] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9292.906037] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9296.034163] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9297.196908] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9299.413493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9299.438554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9299.531471] Lustre: lustre-OST0000: new disk, initializing [ 9299.534419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9299.566612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9301.496667] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9302.262510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9302.265818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9302.293098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9304.549982] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9310.573723] LustreError: 289078:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9310.610550] Lustre: server umount lustre-OST0000 complete [ 9312.224035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9312.226882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9312.231885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9312.739798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9312.746252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9317.856051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9317.859225] Lustre: Skipped 1 previous similar message [ 9321.960814] LustreError: 289279:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9321.964871] LustreError: 289279:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9321.996811] Lustre: server umount lustre-MDT0000 complete [ 9323.360214] LustreError: 286629:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082330 with bad export cookie 15869801083998013607 [ 9323.362880] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9323.364452] LustreError: 286629:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9332.707670] LustreError: 289483:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9332.710196] LustreError: 289483:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9332.759867] Lustre: server umount lustre-MDT0001 complete [ 9336.460330] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9337.509469] Key type lgssc unregistered [ 9337.655379] LNet: 290015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9337.658437] LNetError: 290015:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9337.672438] LNet: Removed LNI 192.168.203.151@tcp [ 9337.988445] Key type .llcrypt unregistered [ 9337.989709] Key type ._llcrypt unregistered [ 9375.220251] Key type ._llcrypt registered [ 9375.221344] Key type .llcrypt registered [ 9375.257317] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9375.356311] Key type .llcrypt unregistered [ 9375.357506] Key type ._llcrypt unregistered [ 9403.497030] Key type ._llcrypt registered [ 9403.498148] Key type .llcrypt registered [ 9403.542389] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9403.656267] Key type .llcrypt unregistered [ 9403.657543] Key type ._llcrypt unregistered [ 9414.643568] Key type ._llcrypt registered [ 9414.645022] Key type .llcrypt registered [ 9414.677132] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9414.774208] Key type .llcrypt unregistered [ 9414.775210] Key type ._llcrypt unregistered [ 9425.957782] Key type ._llcrypt registered [ 9425.960908] Key type .llcrypt registered [ 9426.003768] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9426.097178] Key type .llcrypt unregistered [ 9426.098312] Key type ._llcrypt unregistered [ 9454.689372] Key type ._llcrypt registered [ 9454.691729] Key type .llcrypt registered [ 9454.735725] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9454.831425] Key type .llcrypt unregistered [ 9454.832861] Key type ._llcrypt unregistered [ 9466.569341] Key type ._llcrypt registered [ 9466.570860] Key type .llcrypt registered [ 9466.607489] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9466.701545] Key type .llcrypt unregistered [ 9466.703441] Key type ._llcrypt unregistered [ 9478.173852] Key type ._llcrypt registered [ 9478.175507] Key type .llcrypt registered [ 9478.215184] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9478.328406] Key type .llcrypt unregistered [ 9478.329532] Key type ._llcrypt unregistered [ 9489.614167] Key type ._llcrypt registered [ 9489.615217] Key type .llcrypt registered [ 9489.655025] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9489.758271] Key type .llcrypt unregistered [ 9489.759605] Key type ._llcrypt unregistered [ 9500.937508] Key type ._llcrypt registered [ 9500.938621] Key type .llcrypt registered [ 9500.975352] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9501.085198] Key type .llcrypt unregistered [ 9501.086873] Key type ._llcrypt unregistered [ 9513.110671] Key type ._llcrypt registered [ 9513.112455] Key type .llcrypt registered [ 9513.151980] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9513.251722] Key type .llcrypt unregistered [ 9513.255100] Key type ._llcrypt unregistered [ 9534.982448] Key type ._llcrypt registered [ 9534.983526] Key type .llcrypt registered [ 9535.024782] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9535.142366] Key type .llcrypt unregistered [ 9535.143926] Key type ._llcrypt unregistered [ 9546.568672] Key type ._llcrypt registered [ 9546.571292] Key type .llcrypt registered [ 9546.618383] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9546.731460] Key type .llcrypt unregistered [ 9546.733197] Key type ._llcrypt unregistered [ 9558.069654] Key type ._llcrypt registered [ 9558.071009] Key type .llcrypt registered [ 9558.112521] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9558.232800] Key type .llcrypt unregistered [ 9558.234519] Key type ._llcrypt unregistered [ 9579.011233] Key type ._llcrypt registered [ 9579.012129] Key type .llcrypt registered [ 9579.047786] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 9585.694596] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9585.986836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9586.010871] alg: No test for adler32 (adler32-zlib) [ 9586.875882] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 9586.972762] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 9588.559182] Key type lgssc registered [ 9588.911249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9591.777716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9594.058201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9596.077753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9598.175789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9598.728171] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9367 sec ========== 06:50:06 (1769082606) [ 9599.283191] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:50:06 (1769082606) === [ 9600.339876] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:50:08 (1769082608) === [ 9612.522541] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9613.456184] Key type lgssc unregistered [ 9613.575333] LNet: 311719:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9613.577227] LNetError: 311719:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9613.586278] LNet: Removed LNI 192.168.203.151@tcp [ 9613.851516] Key type .llcrypt unregistered [ 9613.852723] Key type ._llcrypt unregistered