[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 457249042 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003184] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005021] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010149] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.012033] SELinux: Initializing. [ 0.013075] *** VALIDATE selinux *** [ 0.022231] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027937] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032125] *** VALIDATE tmpfs *** [ 0.033551] *** VALIDATE proc *** [ 0.035043] *** VALIDATE cgroup *** [ 0.035968] *** VALIDATE cgroup2 *** [ 0.037266] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040028] Spectre V2 : User space: Vulnerable [ 0.041006] Speculative Store Bypass: Vulnerable [ 0.044319] debug: unmapping init [mem 0xffffffffa5a59000-0xffffffffa5a60fff] [ 0.046251] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047693] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048025] ... version: 2 [ 0.049011] ... bit width: 48 [ 0.050012] ... generic registers: 4 [ 0.051013] ... value mask: 0000ffffffffffff [ 0.052019] ... max period: 00007fffffffffff [ 0.053018] ... fixed-purpose events: 3 [ 0.054009] ... event mask: 000000070000000f [ 0.055252] rcu: Hierarchical SRCU implementation. [ 0.057090] smp: Bringing up secondary CPUs ... [ 0.058384] x86: Booting SMP configuration: [ 0.059027] .... node #0, CPUs: #1 #2 #3 [ 0.063170] smp: Brought up 1 node, 4 CPUs [ 0.065013] smpboot: Max logical packages: 1 [ 0.066013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.217637] node 0 deferred pages initialised in 149ms [ 0.220134] devtmpfs: initialized [ 0.221271] x86/mm: Memory block size: 128MB [ 0.225145] gcov: version magic: 0x41383552 [ 0.228302] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.229135] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.230328] pinctrl core: initialized pinctrl subsystem [ 0.232192] [ 0.232723] ************************************************************* [ 0.234013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.236010] ** ** [ 0.237007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.238010] ** ** [ 0.240010] ** This means that this kernel is built to expose internal ** [ 0.241010] ** IOMMU data structures, which may compromise security on ** [ 0.243008] ** your system. ** [ 0.245011] ** ** [ 0.247013] ** If you see this message and you are not debugging the ** [ 0.248009] ** kernel, report this immediately to your vendor! ** [ 0.250011] ** ** [ 0.251009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.252011] ************************************************************* [ 0.254574] NET: Registered protocol family 16 [ 0.255340] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.257038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.260071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.262404] cpuidle: using governor menu [ 0.264704] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.267462] PCI: Using configuration type 1 for base access [ 0.269132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.279112] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.280015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.283102] cryptd: max_cpu_qlen set to 1000 [ 0.285304] ACPI: Added _OSI(Module Device) [ 0.286026] ACPI: Added _OSI(Processor Device) [ 0.287019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.288042] ACPI: Added _OSI(Processor Aggregator Device) [ 0.292000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.297352] ACPI: Interpreter enabled [ 0.298000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.300012] ACPI: Using IOAPIC for interrupt routing [ 0.302094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.304374] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.312294] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.314044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.317020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.320085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.325360] acpiphp: Slot [2] registered [ 0.326101] acpiphp: Slot [5] registered [ 0.328099] acpiphp: Slot [6] registered [ 0.329109] acpiphp: Slot [7] registered [ 0.331110] acpiphp: Slot [8] registered [ 0.332111] acpiphp: Slot [9] registered [ 0.333136] acpiphp: Slot [10] registered [ 0.335160] acpiphp: Slot [3] registered [ 0.336064] acpiphp: Slot [4] registered [ 0.336940] acpiphp: Slot [11] registered [ 0.338093] acpiphp: Slot [12] registered [ 0.339100] acpiphp: Slot [13] registered [ 0.341123] acpiphp: Slot [14] registered [ 0.342090] acpiphp: Slot [15] registered [ 0.344093] acpiphp: Slot [16] registered [ 0.345079] acpiphp: Slot [17] registered [ 0.346030] acpiphp: Slot [18] registered [ 0.347083] acpiphp: Slot [19] registered [ 0.348094] acpiphp: Slot [20] registered [ 0.350109] acpiphp: Slot [21] registered [ 0.351089] acpiphp: Slot [22] registered [ 0.352085] acpiphp: Slot [23] registered [ 0.353134] acpiphp: Slot [24] registered [ 0.354082] acpiphp: Slot [25] registered [ 0.356066] acpiphp: Slot [26] registered [ 0.357061] acpiphp: Slot [27] registered [ 0.358059] acpiphp: Slot [28] registered [ 0.359083] acpiphp: Slot [29] registered [ 0.360101] acpiphp: Slot [30] registered [ 0.361062] acpiphp: Slot [31] registered [ 0.362068] PCI host bridge to bus 0000:00 [ 0.363013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.365017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.366016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.368015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.370022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.372019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.373132] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.374872] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.377009] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.385021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.388136] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.390016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.391097] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.393014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.395595] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.398693] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.400037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.402841] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.408018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.420017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.426026] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.432200] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.437016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.443018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.456013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.465281] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.470014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.475016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.488018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.498866] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.505016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.511014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.527013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.535984] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.541019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.546016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.558016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.565838] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.571019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.576014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.587017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.597233] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.608016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.614017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.636017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.646293] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.649363] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.652452] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.654365] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.656202] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.662042] iommu: Default domain type: Passthrough [ 0.663425] SCSI subsystem initialized [ 0.664150] ACPI: bus type USB registered [ 0.666127] usbcore: registered new interface driver usbfs [ 0.668177] usbcore: registered new interface driver hub [ 0.669069] usbcore: registered new device driver usb [ 0.671198] pps_core: LinuxPPS API ver. 1 registered [ 0.673013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.676065] PTP clock support registered [ 0.678064] EDAC MC: Ver: 3.0.0 [ 0.681216] PCI: Using ACPI for IRQ routing [ 0.683083] NetLabel: Initializing [ 0.684009] NetLabel: domain hash size = 128 [ 0.685013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.686112] NetLabel: unlabeled traffic allowed by default [ 0.688165] vgaarb: loaded [ 0.690268] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.691012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.698193] clocksource: Switched to clocksource kvm-clock [ 0.800491] VFS: Disk quotas dquot_6.6.0 [ 0.801668] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.804013] *** VALIDATE ramfs *** [ 0.804900] *** VALIDATE hugetlbfs *** [ 0.806015] pnp: PnP ACPI init [ 0.808178] pnp: PnP ACPI: found 6 devices [ 0.823592] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.826218] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.827662] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.829394] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.831135] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.832732] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.834850] NET: Registered protocol family 2 [ 0.836783] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.840369] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.843064] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.847937] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.850248] TCP: Hash tables configured (established 65536 bind 65536) [ 0.852486] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.854404] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.856042] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.857676] NET: Registered protocol family 1 [ 0.859583] RPC: Registered named UNIX socket transport module. [ 0.861349] RPC: Registered udp transport module. [ 0.862412] RPC: Registered tcp transport module. [ 0.864025] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.866113] NET: Registered protocol family 44 [ 0.867165] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.868454] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.869928] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.871260] PCI: CLS 0 bytes, default 64 [ 0.872244] Unpacking initramfs... [ 2.228900] debug: unmapping init [mem 0xffff92b6bcc54000-0xffff92b6bffbffff] [ 2.232473] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.234818] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.237015] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.744109] Initialise system trusted keyrings [ 2.745260] Key type blacklist registered [ 2.748041] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.756927] zbud: loaded [ 2.759364] *** VALIDATE nfs *** [ 2.760291] *** VALIDATE nfs4 *** [ 2.761402] pstore: using deflate compression [ 2.764085] Platform Keyring initialized [ 2.846696] NET: Registered protocol family 38 [ 2.848048] Key type asymmetric registered [ 2.849138] Asymmetric key parser 'x509' registered [ 2.850366] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.852673] io scheduler mq-deadline registered [ 2.853949] io scheduler kyber registered [ 2.855117] io scheduler bfq registered [ 2.856975] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.859842] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.861824] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.863919] ACPI: Power Button [PWRF] [ 2.868557] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.874486] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.884328] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.891112] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.903746] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.930252] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.958776] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.964073] Non-volatile memory driver v1.3 [ 2.965444] Linux agpgart interface v0.103 [ 2.993237] virtio_blk virtio1: [vda] 134024 512-byte logical blocks (68.6 MB/65.4 MiB) [ 2.995378] vda: detected capacity change from 0 to 68620288 [ 3.006941] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.009993] vdb: detected capacity change from 0 to 1073741824 [ 3.025264] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.028267] vdc: detected capacity change from 0 to 2621440000 [ 3.041945] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.044894] vdd: detected capacity change from 0 to 2621440000 [ 3.060223] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.063392] vde: detected capacity change from 0 to 4294967296 [ 3.078732] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.081691] vdf: detected capacity change from 0 to 4294967296 [ 3.093561] libphy: Fixed MDIO Bus: probed [ 3.103073] usbcore: registered new interface driver usbserial_generic [ 3.105330] usbserial: USB Serial support registered for generic [ 3.107607] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.111568] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.112890] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.115094] mousedev: PS/2 mouse device common for all mice [ 3.117342] rtc_cmos 00:05: RTC can wake from S4 [ 3.122315] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.122550] rtc_cmos 00:05: registered as rtc0 [ 3.126255] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.128448] intel_pstate: CPU model not supported [ 3.130785] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.132762] hid: raw HID events driver (C) Jiri Kosina [ 3.135212] usbcore: registered new interface driver usbhid [ 3.136757] usbhid: USB HID core driver [ 3.137912] drop_monitor: Initializing network drop monitor service [ 3.139552] Initializing XFRM netlink socket [ 3.139701] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.140958] NET: Registered protocol family 10 [ 3.146114] Segment Routing with IPv6 [ 3.147530] NET: Registered protocol family 17 [ 3.149526] mpls_gso: MPLS GSO support [ 3.155837] RAS: Correctable Errors collector initialized. [ 3.157982] AVX version of gcm_enc/dec engaged. [ 3.159583] AES CTR mode by8 optimization enabled [ 3.233266] sched_clock: Marking stable (3233245626, 0)->(4027653323, -794407697) [ 3.236320] registered taskstats version 1 [ 3.237597] Loading compiled-in X.509 certificates [ 3.238979] zswap: loaded using pool lzo/zbud [ 3.259519] Key type big_key registered [ 3.271431] Key type encrypted registered [ 3.273304] ima: No TPM chip found, activating TPM-bypass! [ 3.275040] ima: Allocated hash algorithm: sha1 [ 3.276188] ima: No architecture policies found [ 3.277633] evm: Initialising EVM extended attributes: [ 3.279252] evm: security.selinux [ 3.280121] evm: security.ima [ 3.280721] evm: security.capability [ 3.281652] evm: HMAC attrs: 0x1 [ 3.283772] rtc_cmos 00:05: setting system clock to 2026-01-02 02:10:14 UTC (1767319814) [ 3.289953] debug: unmapping init [mem 0xffffffffa6a03000-0xffffffffa6bfffff] [ 3.292890] debug: unmapping init [mem 0xffffffffa5782000-0xffffffffa5a58fff] [ 3.302065] Write protecting the kernel read-only data: 28672k [ 3.305446] debug: unmapping init [mem 0xffffffffa3e03000-0xffffffffa3ffffff] [ 3.307362] debug: unmapping init [mem 0xffffffffa4714000-0xffffffffa47fffff] [ 3.333733] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.341389] systemd[1]: Detected virtualization kvm. [ 3.342814] systemd[1]: Detected architecture x86-64. [ 3.344247] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.368845] systemd[1]: No hostname configured. [ 3.370372] systemd[1]: Set hostname to . [ 3.371849] random: systemd: uninitialized urandom read (16 bytes read) [ 3.373497] systemd[1]: Initializing machine ID from random generator. [ 3.409664] random: ln: uninitialized urandom read (6 bytes read) [ 3.485351] random: systemd: uninitialized urandom read (16 bytes read) [ 3.487751] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.491939] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.496799] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.094750] device-mapper: uevent: version 1.0.3 [ 4.096795] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.801886] virtio_net virtio0 ens2: renamed from eth0 [ 4.827980] random: fast init done [ 4.860638] scsi host0: ata_piix [ 4.881574] scsi host1: ata_piix [ 4.882694] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.884191] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.152178] dracut-initqueue[593]: RTNETLINK answers: File exists [ 9.787909] random: crng init done [ 9.789242] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.070896] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.139256] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.395305] SELinux: Disabled at runtime. [ 11.460675] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.469130] systemd[1]: Detected virtualization kvm. [ 11.470922] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.898979] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.902731] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.905995] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.908719] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.910977] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.917183] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.920920] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slic[ 11.963625] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS e system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.284205] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.510935] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.525237] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.601824] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.624343] EDAC sbridge: Ver: 1.1.2 [ 14.111089] Key type dns_resolver registered [ 14.398629] NFS: Registering the id_resolver key type [ 14.400570] Key type id_resolver registered [ 14.402393] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg217-server login: [ 50.703848] libcfs: loading out-of-tree module taints kernel. [ 50.741241] Key type ._llcrypt registered [ 50.743571] Key type .llcrypt registered [ 50.860545] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 96.575777] hrtimer: interrupt took 22764896 ns [ 125.088422] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 127.275120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 127.324794] alg: No test for adler32 (adler32-zlib) [ 128.955934] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 129.958169] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 131.695168] Key type lgssc registered [ 133.247809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 163.493037] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 164.145470] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 164.446674] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 165.310849] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 165.650335] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 235.922421] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 257.966243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 258.040909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 258.068660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 259.334120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 259.387909] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 259.501270] Lustre: lustre-MDT0000: new disk, initializing [ 259.614933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 259.633568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 263.818305] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 270.807562] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 285.036257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 285.113459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 285.186626] Lustre: 14442:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 285.218517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 285.221315] Lustre: Skipped 1 previous similar message [ 285.300824] Lustre: lustre-MDT0001: new disk, initializing [ 285.376986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 285.395964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 285.404201] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 290.286298] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 298.728886] Lustre: 16502:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 315.488596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 315.547868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 315.724990] Lustre: lustre-OST0000: new disk, initializing [ 315.734734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 315.830467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 321.092499] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 323.601764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 323.608551] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 323.691736] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 339.197853] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 339.257418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 339.340694] Lustre: lustre-OST0001: new disk, initializing [ 339.345159] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 339.391758] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 344.815313] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 347.659764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 347.666245] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 347.733439] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 356.308177] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 364.659638] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 375.873043] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing check_logdir /tmp/testlogs/ [ 380.053287] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing yml_node [ 388.276152] Lustre: DEBUG MARKER: Client: 2.17.0.RC4 [ 391.136799] Lustre: DEBUG MARKER: MDS: 2.17.0.RC4 [ 393.844403] Lustre: DEBUG MARKER: OSS: 2.17.0.RC4 [ 395.133809] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jan 1 21:16:45 EST 2026 [ 412.675904] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 414.033966] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 429.548026] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 429.561190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 429.563615] Lustre: Skipped 2 previous similar messages [ 429.586305] Lustre: Skipped 2 previous similar messages [ 434.145971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 434.152744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 434.172565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 435.522606] LustreError: 21893:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 435.768358] Lustre: server umount lustre-MDT0000 complete [ 443.617835] LustreError: 14435:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320254 with bad export cookie 17498556098162409124 [ 443.620970] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 443.624367] LustreError: 14435:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 444.904397] LustreError: 14450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.919482] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 444.919828] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 444.919832] Lustre: Skipped 3 previous similar messages [ 444.921637] LustreError: 14450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 444.955067] Lustre: Skipped 1 previous similar message [ 448.927640] LustreError: 17810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 448.933277] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 448.938353] LustreError: 17810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 448.943510] Lustre: Skipped 2 previous similar messages [ 449.861839] LustreError: 22343:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 449.876165] LustreError: 22343:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 450.118571] Lustre: server umount lustre-MDT0001 complete [ 460.914203] LustreError: 22941:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 460.921110] LustreError: 22941:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 461.002936] Lustre: server umount lustre-OST0000 complete [ 467.787323] LustreError: 23391:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 467.789913] LustreError: 23391:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 467.980944] Lustre: server umount lustre-OST0001 complete [ 480.077634] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 533.208212] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 549.195987] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 549.235288] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 550.569210] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 553.419538] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 566.995709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 567.062396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.321893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 567.364750] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 567.448346] Lustre: lustre-MDT0000: new disk, initializing [ 567.547798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.583286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 570.052855] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 587.482301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 587.549461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 587.619515] Lustre: 30594:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 587.674474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 587.679685] Lustre: Skipped 1 previous similar message [ 587.749650] Lustre: lustre-MDT0001: new disk, initializing [ 587.883330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 587.935306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 587.955697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 592.394718] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 599.993879] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 603.247348] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 615.529878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 615.592527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 615.832661] Lustre: lustre-OST0000: new disk, initializing [ 615.838346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 615.930678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 617.084420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 617.094434] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 617.135582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 620.972325] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 627.917850] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 632.174096] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 632.411855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 636.879923] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 637.061307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 643.053799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 643.064478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 643.067468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 643.071818] Lustre: Skipped 1 previous similar message [ 644.666140] LustreError: 34225:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 644.668742] LustreError: 34225:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 644.819795] Lustre: server umount lustre-OST0000 complete [ 649.201303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 649.206766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 649.218032] Lustre: Skipped 1 previous similar message [ 653.283079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 653.287953] Lustre: Skipped 2 previous similar messages [ 655.339970] LustreError: 34477:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 655.350313] LustreError: 34477:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 655.477368] Lustre: server umount lustre-MDT0000 complete [ 658.408978] LustreError: 30604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 658.425648] LustreError: 30604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 659.688040] LustreError: 30585:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320470 with bad export cookie 17498556098162410678 [ 659.703542] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.980522] Lustre: server umount lustre-MDT0001 complete [ 665.709953] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 21:21:15 (1767320475) [ 693.702119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.183867] LustreError: 36490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 694.281132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 698.287377] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 699.370622] LustreError: 36490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 709.600374] LustreError: 36490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 709.661158] LustreError: 36490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 712.621802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 719.056740] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 725.180241] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 729.041933] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 733.278434] Lustre: Failing over lustre-MDT0000 [ 733.296671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 733.307851] Lustre: Skipped 2 previous similar messages [ 733.307933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 733.419957] LustreError: 38169:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 733.424416] LustreError: 38169:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 733.493045] Lustre: server umount lustre-MDT0000 complete [ 738.474207] LustreError: 36486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 738.497634] LustreError: 36486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 754.664911] Lustre: 6819:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767320549/real 1767320549] req@ffff92b73adae300 x1853169274617856/t0(0) o400->MGC192.168.202.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767320565 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 754.703511] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 768.884830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 774.112507] LustreError: 37394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 774.126201] LustreError: 37394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 779.564620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 779.571113] Lustre: Skipped 1 previous similar message [ 779.614575] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 783.661210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 783.664214] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 783.716454] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 784.009862] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 793.711821] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 799.201919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 799.222215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 799.236713] Lustre: Skipped 2 previous similar messages [ 804.305319] LustreError: 40213:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 804.347664] LustreError: 40213:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 804.565986] Lustre: server umount lustre-MDT0000 complete [ 810.397312] LustreError: 36469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320621 with bad export cookie 17498556098162412092 [ 810.423690] LustreError: 36469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 810.427663] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 810.959876] Lustre: server umount lustre-MDT0001 complete [ 825.023814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 830.229179] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 840.165815] LustreError: 41151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 840.203911] LustreError: 41151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 842.215894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 846.663601] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 852.140671] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 855.478805] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 860.080274] Lustre: Failing over lustre-MDT0000 [ 860.310556] Lustre: server umount lustre-MDT0000 complete [ 862.177764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 862.190101] Lustre: Skipped 1 previous similar message [ 878.561362] Lustre: 6817:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767320673/real 1767320673] req@ffff92b6067f1c00 x1853169274648064/t0(0) o400->MGC192.168.202.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767320689 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 878.593066] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 892.470127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 904.576345] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 909.226782] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 909.798382] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 909.806233] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 909.820437] Lustre: Skipped 1 previous similar message [ 909.851749] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 920.408214] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 929.252240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 929.259724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 929.283295] Lustre: Skipped 2 previous similar messages [ 929.289899] Lustre: Skipped 2 previous similar messages [ 931.715584] LustreError: 44714:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 931.721827] LustreError: 44714:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 931.889163] Lustre: server umount lustre-MDT0000 complete [ 936.856043] LustreError: 41112:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320748 with bad export cookie 17498556098162413023 [ 936.862395] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 936.872302] LustreError: 41112:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 937.320505] Lustre: server umount lustre-MDT0001 complete [ 946.259708] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 21:25:55 (1767320755) [ 979.770671] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 1045.686742] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1066.868934] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1067.177808] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1068.605401] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1071.317980] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1136.727567] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1154.584862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1154.657669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1154.918671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1154.938560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1154.996799] Lustre: lustre-MDT0000: new disk, initializing [ 1155.069138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1155.072421] Lustre: Skipped 3 previous similar messages [ 1155.081822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1160.574516] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1179.074547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1179.181758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1179.269666] Lustre: 57033:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1179.487179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1183.749710] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1191.354921] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1195.087686] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1207.799112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1207.873283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1209.505844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1209.543300] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1209.589704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1209.600273] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 1214.571863] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1222.653407] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1228.504910] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1250.795644] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1250.812582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.840963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1250.851115] Lustre: Skipped 2 previous similar messages [ 1262.047197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1262.138024] LustreError: 60384:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1262.140541] LustreError: 60384:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1262.312334] Lustre: server umount lustre-OST0000 complete [ 1273.714144] Lustre: server umount lustre-MDT0000 complete [ 1276.394735] LustreError: 57043:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1276.416549] LustreError: 57043:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 1278.084637] LustreError: 57025:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767321089 with bad export cookie 17498556098162413422 [ 1278.086114] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1278.100127] LustreError: 57025:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1278.620941] Lustre: server umount lustre-MDT0001 complete [ 1292.091450] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 1295.279988] Key type lgssc unregistered [ 1295.655632] LNet: 61562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1295.663379] LNetError: 61562:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1296.698064] LNet: Removed LNI 192.168.202.117@tcp [ 1297.892164] Key type .llcrypt unregistered [ 1297.897822] Key type ._llcrypt unregistered [ 1385.162713] Key type ._llcrypt registered [ 1385.166666] Key type .llcrypt registered [ 1385.317822] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1386.584650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1386.603485] alg: No test for adler32 (adler32-zlib) [ 1387.778108] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 1387.999621] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 1389.681561] Key type lgssc registered [ 1390.773047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1399.132794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1399.160867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1400.765037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1405.035063] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1410.019300] LustreError: 66017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1410.055623] LustreError: 66017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1415.148166] LustreError: 66017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1417.614148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1418.075925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1422.495717] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1428.628972] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1432.622439] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1442.249351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1442.557392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1443.570423] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1445.650943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1447.484513] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1452.803259] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1467.544358] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 21:34:37 (1767321277) [ 1469.571595] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1471.463200] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1479.536688] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 21:34:49 (1767321289) [ 1489.570747] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 21:34:59 (1767321299) [ 1490.443853] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1499.135408] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1499.139832] Lustre: Skipped 2 previous similar messages [ 1501.235275] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 21:35:10 (1767321310) [ 1503.486784] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1503.491264] Lustre: Skipped 2 previous similar messages [ 1512.214395] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1512.220793] Lustre: Skipped 4 previous similar messages [ 1528.224659] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 21:35:37 (1767321337) [ 1531.370924] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1535.358293] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1545.102545] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 21:35:54 (1767321354) [ 1547.100447] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1547.106935] Lustre: Skipped 6 previous similar messages [ 1562.056390] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 21:36:11 (1767321371) [ 1576.937459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1576.949959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1576.972917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1578.975965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1578.980797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1578.997047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1582.610225] LustreError: 72961:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1582.818512] Lustre: server umount lustre-OST0000 complete [ 1589.233300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1589.243877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1589.248865] Lustre: Skipped 1 previous similar message [ 1589.275994] Lustre: Skipped 2 previous similar messages [ 1594.337326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1595.274765] LustreError: 73211:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1595.277304] LustreError: 73211:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1595.475089] Lustre: server umount lustre-MDT0000 complete [ 1599.459098] LustreError: 66017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1599.490603] LustreError: 66017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1601.639817] LustreError: 65997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767321412 with bad export cookie 4715330765344476232 [ 1601.646619] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1601.659484] LustreError: 65997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1601.875861] LustreError: 73460:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1601.884874] LustreError: 73460:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1602.059464] Lustre: server umount lustre-MDT0001 complete [ 1616.279312] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 1618.979147] Key type lgssc unregistered [ 1619.246721] LNet: 74137:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1619.258946] LNetError: 74137:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1619.284564] LNet: Removed LNI 192.168.202.117@tcp [ 1619.986651] Key type .llcrypt unregistered [ 1619.990644] Key type ._llcrypt unregistered [ 1655.543613] Key type ._llcrypt registered [ 1655.545823] Key type .llcrypt registered [ 1655.641959] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 1714.829578] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1715.759279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1715.771699] alg: No test for adler32 (adler32-zlib) [ 1716.776631] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 1717.032731] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 1718.719158] Key type lgssc registered [ 1719.597592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1737.030253] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1737.329586] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1738.614645] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1741.062226] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1755.738481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1755.812774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1755.836252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1757.164117] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1757.190712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1757.283597] Lustre: lustre-MDT0000: new disk, initializing [ 1757.374315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.394426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1761.367429] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1776.879665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1776.959101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1777.086060] Lustre: 82952:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1777.145235] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1777.150088] Lustre: Skipped 1 previous similar message [ 1777.236900] Lustre: lustre-MDT0001: new disk, initializing [ 1777.346794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1777.395092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1777.402216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1781.418449] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1788.428240] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1792.089236] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1804.513785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1804.574693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1804.733720] Lustre: lustre-OST0000: new disk, initializing [ 1804.739089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1804.805847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1810.274248] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1810.488289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1810.495866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1810.566606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1817.425291] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1821.639448] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1821.787400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1825.968960] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1826.124265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1831.392899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1831.394587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1831.403973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1831.422227] Lustre: Skipped 1 previous similar message [ 1836.516923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1836.518880] Lustre: Skipped 2 previous similar messages [ 1841.637681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1841.643321] Lustre: Skipped 1 previous similar message [ 1842.143119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1842.171682] LustreError: 86585:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1842.228767] Lustre: server umount lustre-OST0000 complete [ 1849.312315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1849.327080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1849.341229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1851.890366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.191683] LustreError: 86837:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1853.202582] LustreError: 86837:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1853.320616] Lustre: server umount lustre-MDT0000 complete [ 1858.587722] LustreError: 82945:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767321669 with bad export cookie 1672883038697010427 [ 1858.606763] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1858.616268] LustreError: 82945:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1858.807205] LustreError: 87086:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1858.816958] LustreError: 87086:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1859.042695] Lustre: server umount lustre-MDT0001 complete [ 1871.976256] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 21:41:20 (1767321680) [ 1881.387861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1881.852458] LustreError: 87998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1881.873757] LustreError: 87998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1881.927753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1886.537549] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1887.201732] LustreError: 87998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1892.325555] LustreError: 87998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.474653] LustreError: 87998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1898.594941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1898.895233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1902.972903] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1908.841775] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1912.745080] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1922.726128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1923.004980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.934042] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1933.965458] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1938.303714] LustreError: 90580:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1939.187334] Lustre: Modifying parameter general.timeout=1 in log params [ 1940.989891] Lustre: Modifying parameter general.timeout=3 in log params [ 1940.993038] Lustre: Skipped 1 previous similar message [ 1943.292515] Lustre: Modifying parameter general.timeout=6 in log params [ 1943.297951] Lustre: Skipped 2 previous similar messages [ 1947.861286] Lustre: Modifying parameter general.timeout=12 in log params [ 1947.865856] Lustre: Skipped 5 previous similar messages [ 1955.918523] Lustre: Modifying parameter general.timeout=23 in log params [ 1955.922834] Lustre: Skipped 10 previous similar messages [ 1972.179281] Lustre: Modifying parameter general.timeout=44 in log params [ 1972.184116] Lustre: Skipped 20 previous similar messages [ 2005.151724] Lustre: Modifying parameter general.timeout=82 in log params [ 2005.155926] Lustre: Skipped 37 previous similar messages [ 2057.240199] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 21:44:27 (1767321867) [ 2059.557898] Lustre: 97522:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 2061.333600] Lustre: 97621:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 2064.365508] Lustre: 97830:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 2064.376724] Lustre: 97830:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 2069.263156] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule= in log params [ 2069.275221] Lustre: Skipped 55 previous similar messages [ 2091.465511] Lustre: 99396:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 2091.473706] Lustre: 99396:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 2091.493093] LustreError: 99396:0:(mgs_handler.c:1129:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 2095.469336] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 21:45:05 (1767321905) [ 2107.360955] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2107.367886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2107.378603] Lustre: Skipped 1 previous similar message [ 2107.389261] Lustre: Skipped 4 previous similar messages [ 2113.337292] LustreError: 100096:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2113.339920] LustreError: 100096:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2113.406939] Lustre: server umount lustre-MDT0000 complete [ 2116.976325] LustreError: 87978:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767321928 with bad export cookie 1672883038697011309 [ 2116.994268] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2117.270850] Lustre: server umount lustre-MDT0001 complete [ 2136.596195] LustreError: 100644:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2136.612142] LustreError: 100644:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2136.710675] Lustre: server umount lustre-OST0000 complete [ 2147.042667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2150.959873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2156.133644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2160.676520] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2172.689581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2172.910102] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2173.116877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2177.910906] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2194.210476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2194.358719] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2194.429075] Lustre: 102768:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2198.813789] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2203.567781] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2207.712277] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2219.338306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2219.494473] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2219.785556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2219.789915] Lustre: Skipped 1 previous similar message [ 2225.795332] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2231.630708] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2233.348446] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 2233.365980] Lustre: Skipped 15 previous similar messages [ 2233.433414] Lustre: 105391:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2240.997920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2241.005513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2241.015771] Lustre: Skipped 1 previous similar message [ 2241.024621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2241.029241] Lustre: Skipped 3 previous similar messages [ 2246.746695] LustreError: 105651:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2246.754970] LustreError: 105651:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2247.005781] Lustre: server umount lustre-OST0000 complete [ 2256.354737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2256.371766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2256.398665] Lustre: Skipped 1 previous similar message [ 2258.231871] Lustre: server umount lustre-MDT0000 complete [ 2261.988502] LustreError: 104842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2262.010610] LustreError: 104842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2264.011370] LustreError: 104796:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322075 with bad export cookie 1672883038697014312 [ 2264.024467] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2264.026408] LustreError: 104796:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2264.271468] LustreError: 106152:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2264.276257] LustreError: 106152:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2264.597765] Lustre: server umount lustre-MDT0001 complete [ 2279.870798] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 2283.266832] Key type lgssc unregistered [ 2283.613373] LNet: 106827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2283.618392] LNetError: 106827:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2283.658144] LNet: Removed LNI 192.168.202.117@tcp [ 2284.441271] Key type .llcrypt unregistered [ 2284.443254] Key type ._llcrypt unregistered [ 2373.726086] Key type ._llcrypt registered [ 2373.728583] Key type .llcrypt registered [ 2373.904525] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 2374.846385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2374.896852] alg: No test for adler32 (adler32-zlib) [ 2375.970528] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 2376.238078] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 2377.912862] Key type lgssc registered [ 2379.014893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2386.469316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2386.486956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2388.032087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2392.533773] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2398.187104] LustreError: 111284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.199751] LustreError: 111284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2403.296936] LustreError: 111284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.321879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2405.047903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2410.728843] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2418.407838] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2423.301821] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2435.274449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2435.985130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2441.603385] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2447.501589] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2465.386168] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2476.515859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2476.535481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.546539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2479.079131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2479.080366] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.268839] LustreError: 114399:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2480.502926] Lustre: server umount lustre-MDT0000 complete [ 2483.998891] LustreError: 111264:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322295 with bad export cookie 2043035168663484028 [ 2484.009983] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2484.010534] LustreError: 111264:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2484.151824] LustreError: 114603:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2484.163574] LustreError: 114603:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2484.195831] LustreError: 111280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.199992] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.227816] Lustre: Skipped 1 previous similar message [ 2484.536619] Lustre: server umount lustre-MDT0001 complete [ 2490.936503] LustreError: 114949:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2490.945337] LustreError: 114949:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2491.118451] Lustre: server umount lustre-OST0000 complete [ 2502.859856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2506.748845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2511.132034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2515.699984] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2527.608421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.875637] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2527.912422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2528.133205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2531.709776] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2543.010658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2543.154980] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2543.183722] Lustre: 117074:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2543.620072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2548.257245] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2553.950507] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2558.183626] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2569.484805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2569.678883] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2569.936428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2575.249599] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2580.965365] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2583.582112] Lustre: 119701:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2586.028498] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2593.761807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2593.776343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.798488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2593.803081] Lustre: Skipped 2 previous similar messages [ 2596.046860] LustreError: 119959:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2596.052897] LustreError: 119959:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2596.200537] Lustre: server umount lustre-OST0000 complete [ 2604.512919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.533792] Lustre: Skipped 1 previous similar message [ 2604.539338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2604.542931] Lustre: Skipped 3 previous similar messages [ 2606.606601] LustreError: 120210:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2606.608461] LustreError: 120210:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2606.788700] Lustre: server umount lustre-MDT0000 complete [ 2609.632861] LustreError: 117085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.650492] LustreError: 117085:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2610.760627] LustreError: 119703:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322421 with bad export cookie 2043035168663484854 [ 2610.767018] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2610.770043] LustreError: 119703:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2611.018806] Lustre: server umount lustre-MDT0001 complete [ 2623.402864] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 2626.204876] Key type lgssc unregistered [ 2626.432780] LNet: 121136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2626.446571] LNetError: 121136:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2626.477434] LNet: Removed LNI 192.168.202.117@tcp [ 2627.225292] Key type .llcrypt unregistered [ 2627.226995] Key type ._llcrypt unregistered [ 2700.535756] Key type ._llcrypt registered [ 2700.538446] Key type .llcrypt registered [ 2700.631344] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 2701.448068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2701.463946] alg: No test for adler32 (adler32-zlib) [ 2702.517721] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 2702.715668] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 2704.359174] Key type lgssc registered [ 2705.381369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2711.621551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2711.661948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2713.123907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.884749] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2723.299131] LustreError: 125587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.325540] LustreError: 125587:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2727.293907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2727.669626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2730.925492] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2735.657539] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2739.371154] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2748.049173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2748.409636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2753.898873] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2758.753462] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2763.628862] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2764.408238] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2765.762926] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2765.768131] Lustre: Skipped 1 previous similar message [ 2767.840052] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2767.848603] Lustre: Skipped 2 previous similar messages [ 2772.561068] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2772.566450] Lustre: Skipped 5 previous similar messages [ 2781.204595] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2781.208435] Lustre: Skipped 11 previous similar messages [ 2797.890423] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2797.900726] Lustre: Skipped 20 previous similar messages [ 2830.383430] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2830.387841] Lustre: Skipped 40 previous similar messages [ 2847.285881] Lustre: 133733:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2849.613712] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 21:57:39 (1767322659) [ 2850.840802] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2852.176399] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 21:57:42 (1767322662) [ 2854.884896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2854.888407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.900021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2855.904294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2855.907443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2855.910410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2855.927496] Lustre: Skipped 1 previous similar message [ 2860.733088] LustreError: 134028:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2860.810525] Lustre: server umount lustre-OST0000 complete [ 2865.632961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2865.641785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2865.655037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2870.285102] LustreError: 134277:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2870.293800] LustreError: 134277:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2870.379954] Lustre: server umount lustre-MDT0000 complete [ 2871.270046] LustreError: 125588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2871.276202] LustreError: 125588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2873.636147] LustreError: 125568:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322684 with bad export cookie 456395401005179638 [ 2873.638941] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2873.643702] LustreError: 125568:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2873.764760] LustreError: 134527:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2873.770496] LustreError: 134527:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2873.902347] Lustre: server umount lustre-MDT0001 complete [ 2882.613738] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 2884.455457] Key type lgssc unregistered [ 2884.694904] LNet: 135203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2884.706447] LNetError: 135203:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2884.723241] LNet: Removed LNI 192.168.202.117@tcp [ 2885.327098] Key type .llcrypt unregistered [ 2885.328432] Key type ._llcrypt unregistered [ 2888.223592] Key type ._llcrypt registered [ 2888.225819] Key type .llcrypt registered [ 2888.311452] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2891.488247] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules [ 2892.340807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2892.363386] alg: No test for adler32 (adler32-zlib) [ 2893.292776] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 2893.453744] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 2895.079426] Key type lgssc registered [ 2896.104077] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2897.003625] LustreError: 136366:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 2918.708759] LustreError: 136366:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2951.079459] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 2956.010195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2956.023774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2957.356776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2959.858185] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2965.690434] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 21:59:36 (1767322776) [ 2967.524279] LustreError: 140452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2967.546646] LustreError: 140452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2969.545215] LustreError: 140485:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2969.554750] LustreError: 140485:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2969.560395] LustreError: 140485:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 2969.668915] LustreError: 141307:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2969.763978] Lustre: server umount lustre-MDT0000 complete [ 2980.579450] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 2982.179293] Key type lgssc unregistered [ 2982.412039] LNet: 142184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2982.418361] LNetError: 142184:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2982.428916] LNet: Removed LNI 192.168.202.117@tcp [ 2982.933695] Key type .llcrypt unregistered [ 2982.935073] Key type ._llcrypt unregistered [ 3053.229326] Key type ._llcrypt registered [ 3053.230757] Key type .llcrypt registered [ 3053.367204] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 3054.238376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3054.276260] alg: No test for adler32 (adler32-zlib) [ 3055.410516] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 3055.705519] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 3057.419057] Key type lgssc registered [ 3058.353062] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3065.267239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3065.292700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3066.826648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3070.247926] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3076.070911] LustreError: 146453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3076.091590] LustreError: 146453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3079.401752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3080.016459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3083.936793] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3088.780550] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3091.405365] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3100.124431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3100.616689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3106.030169] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3110.663339] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3153.579920] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:02:43 (1767322963) [ 3157.544786] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 3158.751735] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:02:49 (1767322969) [ 3162.084901] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3162.087661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3162.096425] Lustre: Skipped 2 previous similar messages [ 3162.102719] Lustre: Skipped 1 previous similar message [ 3167.200835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3167.210914] Lustre: Skipped 3 previous similar messages [ 3167.987065] LustreError: 149894:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3168.069171] Lustre: server umount lustre-MDT0000 complete [ 3171.105182] LustreError: 146435:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322982 with bad export cookie 4974910683078870607 [ 3171.106241] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3171.113720] LustreError: 146435:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3171.232422] LustreError: 150095:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3171.237203] LustreError: 150095:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3171.331821] Lustre: server umount lustre-MDT0001 complete [ 3187.468559] LustreError: 150441:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3187.473337] LustreError: 150441:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3187.718489] Lustre: server umount lustre-OST0000 complete [ 3205.143911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3205.587179] LustreError: 151553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3205.762379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3209.216271] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3211.234478] LustreError: 151553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3215.342932] LustreError: 151553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.394802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3219.945359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3223.568635] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3227.703130] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3230.260955] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3236.963578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3244.598736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3244.643711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3244.779423] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3244.784665] LustreError: 151540:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3244.804964] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3244.809417] LustreError: 153993:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 3244.812293] LustreError: 153993:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 3244.814872] LustreError: 153993:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 3244.853833] Lustre: server umount lustre-OST0000 complete [ 3244.856244] LustreError: 153993:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3252.871990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3253.036389] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3253.040511] LustreError: 151540:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3253.054667] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3253.062213] LustreError: 154647:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 3253.071255] LustreError: 154647:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 3253.076780] LustreError: 154647:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 3253.128213] Lustre: server umount lustre-OST0000 complete [ 3253.132776] LustreError: 154647:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3255.793273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3263.608619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3263.729936] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3263.735890] Lustre: Found index 0 for lustre-OST0000, updating log [ 3263.742809] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3263.801369] Lustre: lustre-OST0000: new disk, initializing [ 3263.803616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3263.985444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3267.980749] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3270.560709] Lustre: Failing over lustre-OST0000 [ 3270.596229] LustreError: 155970:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3270.600623] LustreError: 155970:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3270.675372] Lustre: server umount lustre-OST0000 complete [ 3274.209590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3274.214600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3274.219373] Lustre: Skipped 1 previous similar message [ 3279.329299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3279.339521] Lustre: Skipped 2 previous similar messages [ 3279.766661] LustreError: 156227:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3279.772301] LustreError: 156227:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3279.868778] Lustre: server umount lustre-MDT0000 complete [ 3282.911182] LustreError: 155429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323094 with bad export cookie 4974910683078873344 [ 3282.918608] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3282.925688] LustreError: 155429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3293.157767] LustreError: 156476:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3293.162485] LustreError: 156476:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3293.258408] Lustre: server umount lustre-MDT0001 complete [ 3299.232981] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:05:09 (1767323109) [ 3362.094607] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 3375.093897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.363655] LustreError: 161213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3375.430166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3378.981875] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3383.219893] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3384.802270] LustreError: 161212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3384.818303] LustreError: 161212:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3390.976408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3394.606136] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3399.064985] Lustre: 162850:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3406.528271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3407.050185] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3411.053769] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3416.312266] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3416.325233] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3416.325941] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3423.220131] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3423.289906] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3423.346065] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3423.454317] Lustre: lustre-OST0001: new disk, initializing [ 3423.457760] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3423.727466] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3423.731098] Lustre: Skipped 2 previous similar messages [ 3428.281393] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3431.010541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3431.026534] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3431.150147] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3434.737441] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3440.680755] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3451.012123] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.17@tcp (stopping) [ 3451.875433] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3451.889175] Lustre: Skipped 2 previous similar messages [ 3455.777861] LustreError: 165575:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3455.780527] LustreError: 165575:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3455.890452] Lustre: server umount lustre-MDT0001 complete [ 3456.115971] LustreError: 162955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3456.124492] LustreError: 162955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3458.829046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3465.687541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3465.754197] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3465.757317] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3465.788581] Lustre: 161199:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3467.634555] LustreError: 166437:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3467.638029] LustreError: 166437:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3467.648422] LustreError: 161204:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3467.713569] LustreError: 166448:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.202.117@tcp: cfg command failed: rc = -17 [ 3467.717601] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3467.717601] [ 3469.305637] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3471.335836] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3476.124659] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 22:08:06 (1767323286) [ 3533.819747] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 3548.851731] Lustre: 171641:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3558.172501] Lustre: 172376:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3579.856789] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3582.637141] Lustre: 173895:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3660.773101] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3660.774111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3660.775776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3660.775801] Lustre: Skipped 3 previous similar messages [ 3660.779928] Lustre: Skipped 3 previous similar messages [ 3663.633669] LustreError: 174347:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3663.640892] LustreError: 174347:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3663.696573] Lustre: server umount lustre-MDT0000 complete [ 3665.889883] LustreError: 165441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3665.896924] LustreError: 165441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 3666.162473] LustreError: 164558:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323477 with bad export cookie 4974910683078874282 [ 3666.165414] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3666.172026] LustreError: 164558:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3666.436904] Lustre: server umount lustre-MDT0001 complete [ 3681.661692] Lustre: server umount lustre-OST0000 complete [ 3694.658229] Lustre: server umount lustre-OST0001 complete [ 3701.801431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3713.702066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3714.351232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3720.551293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3729.846671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3730.703873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3756.137752] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 3770.141269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3770.152295] Lustre: lustre-MDT0000: reset Object Index mappings [ 3770.748748] LustreError: 179997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3770.933754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3770.943878] Lustre: Skipped 1 previous similar message [ 3774.627542] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3779.110756] Lustre: 180556:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3786.841904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3786.853626] Lustre: lustre-MDT0001: reset Object Index mappings [ 3790.790712] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3795.127745] Lustre: 181647:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3802.963380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3808.587248] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3812.776727] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3814.840075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3817.267293] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3819.061554] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3819.074810] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3821.045953] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3826.028571] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3828.135635] Lustre: 183886:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3861.653824] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:14:32 (1767323672) [ 3889.632773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3889.636956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3889.653180] Lustre: Skipped 2 previous similar messages [ 3889.656416] Lustre: Skipped 4 previous similar messages [ 3889.662078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3895.357902] LustreError: 184238:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3895.363190] LustreError: 184238:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3895.407515] Lustre: server umount lustre-MDT0000 complete [ 3899.875467] LustreError: 183915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3899.890658] LustreError: 183915:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 3900.686945] LustreError: 179976:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323711 with bad export cookie 4974910683078921245 [ 3900.693328] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3900.694786] LustreError: 179976:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3904.993630] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3905.002983] Lustre: Skipped 1 previous similar message [ 3907.096134] Lustre: server umount lustre-MDT0001 complete [ 3919.183139] Lustre: server umount lustre-OST0000 complete [ 3938.271162] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3938.410443] Lustre: server umount lustre-OST0001 complete [ 3946.886792] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 3985.912131] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 3996.533144] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3996.622095] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3997.619540] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3999.174856] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4009.583979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4009.625861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4009.751784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4009.776672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4009.838463] Lustre: lustre-MDT0000: new disk, initializing [ 4009.908996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4009.916398] Lustre: Skipped 3 previous similar messages [ 4009.925716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4012.480184] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4017.359439] Lustre: server umount lustre-MDT0000 complete [ 4019.411633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4026.022323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4026.062548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4026.277741] Lustre: Found index 0 for lustre-MDT0000, updating log [ 4026.288273] Lustre: 194308:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4026.300404] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 4028.809965] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4032.436760] LustreError: 194868:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4032.440887] LustreError: 194868:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 4032.626574] Lustre: server umount lustre-MDT0000 complete [ 4038.092628] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:17:28 (1767323848) [ 4039.362589] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 4040.809208] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:17:30 (1767323850) [ 4087.924689] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 4098.717673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4102.073323] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4105.579919] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4110.950313] loop: module loaded [ 4133.352669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4133.401518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4133.579863] Lustre: lustre-OST0001: new disk, initializing [ 4133.583749] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4133.588805] Lustre: Skipped 1 previous similar message [ 4135.215174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:ost [ 4135.221373] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:ost] [ 4135.239921] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x240000400 [ 4136.974958] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4143.170688] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4145.357732] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4339.197262] Lustre: 208881:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4339.201841] Lustre: 208881:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4339.205313] Lustre: 208881:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4339.209454] Lustre: 208881:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4339.213237] Lustre: 208881:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4339.216991] Lustre: 208881:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4454.046130] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:24:24 (1767324264) [ 4468.194561] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4468.216046] Lustre: Skipped 1 previous similar message [ 4468.220305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4468.230045] Lustre: Skipped 10 previous similar messages [ 4469.086058] LustreError: 209089:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4469.091893] LustreError: 209089:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4469.282254] Lustre: server umount lustre-MDT0000 complete [ 4479.635521] LustreError: 203312:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324290 with bad export cookie 4974910683078982026 [ 4479.641668] LustreError: 203312:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4479.641965] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4485.811149] Lustre: server umount lustre-OST0001 complete [ 4492.981071] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 4532.786408] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 4545.187641] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4545.472426] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4546.818963] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4548.322445] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4558.242145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4558.295505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4558.428892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4558.455486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4558.506671] Lustre: lustre-MDT0000: new disk, initializing [ 4558.555515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4558.559135] Lustre: Skipped 3 previous similar messages [ 4558.570990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4560.608822] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4571.633129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4571.696844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4571.759515] Lustre: 217283:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4571.765149] Lustre: 217283:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 4571.793927] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4571.799535] Lustre: Skipped 1 previous similar message [ 4571.850535] Lustre: lustre-MDT0001: new disk, initializing [ 4571.943098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4571.950942] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4574.065355] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4578.585603] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4580.717215] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4588.935083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4588.994396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4589.193466] Lustre: lustre-OST0000: new disk, initializing [ 4589.197068] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4590.567179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4590.580269] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4590.662689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4592.931120] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4597.615224] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4604.305685] Lustre: lustre-MDD0000: changelog on [ 4606.016755] Lustre: lustre-MDD0001: changelog on [ 5713.703365] Lustre: lustre-MDD0001: changelog off [ 5714.528806] Lustre: 223601:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5714.929793] Lustre: lustre-MDD0000: changelog off [ 5717.473412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5717.482460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5717.493662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5721.568287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5721.571302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5723.449394] LustreError: 227680:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5723.452026] LustreError: 227680:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5723.544666] Lustre: server umount lustre-OST0000 complete [ 5727.202396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5727.206195] Lustre: Skipped 4 previous similar messages [ 5732.178991] Lustre: server umount lustre-MDT0000 complete [ 5732.320450] LustreError: 223601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5732.329853] LustreError: 223601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5734.330677] LustreError: 227681:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325545 with bad export cookie 4974910683079648230 [ 5734.334507] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5734.336918] LustreError: 227681:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5734.497676] Lustre: server umount lustre-MDT0001 complete [ 5740.558954] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 5741.943771] Key type lgssc unregistered [ 5742.111904] LNet: 228854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5742.115966] LNetError: 228854:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5742.123616] LNet: Removed LNI 192.168.202.117@tcp [ 5742.503220] Key type .llcrypt unregistered [ 5742.506286] Key type ._llcrypt unregistered [ 5760.955720] Key type ._llcrypt registered [ 5760.956977] Key type .llcrypt registered [ 5761.001191] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 5795.596389] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 5796.180690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5796.193450] alg: No test for adler32 (adler32-zlib) [ 5797.095779] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 5797.203960] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 5798.799185] Key type lgssc registered [ 5799.364846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5809.099286] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5809.192754] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5810.153054] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5811.282295] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5818.948338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5818.968918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5818.979201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5820.097262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5820.115879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5820.187539] Lustre: lustre-MDT0000: new disk, initializing [ 5820.257449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5820.274724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5822.120447] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5831.291593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5831.336478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5831.371285] Lustre: 237709:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5831.397385] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5831.400213] Lustre: Skipped 1 previous similar message [ 5831.437431] Lustre: lustre-MDT0001: new disk, initializing [ 5831.503410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5831.531657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5831.543766] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5833.405332] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5837.687646] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5839.779430] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5846.588862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5846.618870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5846.759319] Lustre: lustre-OST0000: new disk, initializing [ 5846.761669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5846.804366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5850.213671] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5854.902302] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5855.247715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5855.252368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5855.299571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5858.613760] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_module obdecho/obdecho [ 5859.953071] LustreError: 241019:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5859.955565] LustreError: 241019:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 5861.458518] LustreError: 241116:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 5867.490544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5867.498693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5867.512589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5870.566472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5870.566499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5870.572479] Lustre: Skipped 1 previous similar message [ 5870.587413] Lustre: Skipped 1 previous similar message [ 5873.237245] LustreError: 241457:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5873.299545] Lustre: server umount lustre-MDT0000 complete [ 5875.167738] LustreError: 237702:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325686 with bad export cookie 1667256829627921984 [ 5875.168410] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5875.172286] LustreError: 237702:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5875.253238] LustreError: 241660:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5875.255287] LustreError: 241660:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5875.349755] Lustre: server umount lustre-MDT0001 complete [ 5888.697512] LustreError: 242004:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5888.701235] LustreError: 242004:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5888.756791] Lustre: server umount lustre-OST0000 complete [ 5910.110368] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 5939.423122] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 5947.220464] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5947.260837] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5948.111480] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5949.170181] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5956.319040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5956.344864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5956.450215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5956.486602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5956.528743] Lustre: lustre-MDT0000: new disk, initializing [ 5956.557287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5956.564246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5958.229520] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5966.556837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5966.595588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5966.649133] Lustre: 250796:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5966.683506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5966.685989] Lustre: Skipped 1 previous similar message [ 5966.729389] Lustre: lustre-MDT0001: new disk, initializing [ 5966.783590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5966.803801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5966.809135] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5968.963901] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5973.485560] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5975.508943] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5981.627110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5981.659738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5981.776599] Lustre: lustre-OST0000: new disk, initializing [ 5981.779545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5981.815702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5984.565570] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5988.407331] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5989.381314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5989.386251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5989.473529] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5990.402603] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5990.508610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5992.787934] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5992.890995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5994.466944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5994.467454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5994.482809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5994.488714] Lustre: Skipped 1 previous similar message [ 5995.890270] LustreError: 254427:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5995.893701] LustreError: 254427:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5995.952713] Lustre: server umount lustre-OST0000 complete [ 5999.585339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5999.586367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5999.590955] Lustre: Skipped 2 previous similar messages [ 6004.425243] LustreError: 254677:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6004.430591] LustreError: 254677:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6004.498063] Lustre: server umount lustre-MDT0000 complete [ 6006.706500] LustreError: 250788:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325817 with bad export cookie 1667256829627922950 [ 6006.707912] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6006.711603] LustreError: 250788:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6006.869229] Lustre: server umount lustre-MDT0001 complete [ 6008.555595] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 22:50:19 (1767325819) [ 6009.275387] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6010.179877] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 22:50:20 (1767325820) [ 6015.919222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6016.096988] LustreError: 255933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6016.103572] LustreError: 255933:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6016.123980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6017.780574] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6021.600360] LustreError: 255933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6022.959724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6024.924748] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6027.428945] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6029.185576] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6033.788433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6036.717692] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6039.081690] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6053.857987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6053.858379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6053.862237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6053.862243] Lustre: Skipped 1 previous similar message [ 6053.864541] Lustre: Skipped 1 previous similar message [ 6053.867131] LustreError: Skipped 1 previous similar message [ 6056.014599] LustreError: 258633:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6056.021412] LustreError: 258633:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6056.116407] Lustre: server umount lustre-MDT0000 complete [ 6058.167563] LustreError: 255912:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325869 with bad export cookie 1667256829627923832 [ 6058.173935] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6058.179314] LustreError: 255912:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6058.975887] LustreError: 258485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6064.095931] LustreError: 255927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6064.096142] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6064.111547] LustreError: 255927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6064.118849] Lustre: Skipped 5 previous similar messages [ 6064.412576] Lustre: server umount lustre-MDT0001 complete [ 6067.848950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6075.662042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6075.771518] LustreError: 260230:0:(llog.c:1643:llog_backup()) MGC192.168.202.117@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6075.777498] Lustre: 260230:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.202.117@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6080.626490] LustreError: 260238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6088.672114] LustreError: 260238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6088.680191] LustreError: 260238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6106.231065] LustreError: 260236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6106.238049] LustreError: 260236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 6136.800043] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6136.802515] Lustre: 260230:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.202.117@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6138.848821] LustreError: 260265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6138.857198] LustreError: 260265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [ 6162.400946] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6162.411355] Lustre: 255924:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.202.117@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6177.254161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6177.257728] Lustre: Skipped 2 previous similar messages [ 6178.278493] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6178.864531] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6182.879345] LustreError: 255924:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6183.426264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6202.847770] LustreError: 234185:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cc453412 x1853175216843520/t0(0) o250->MGC192.168.202.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6202.857129] LustreError: 260265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6202.858272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6202.863765] LustreError: 260265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 47 previous similar messages [ 6204.398967] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6206.267446] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6206.846219] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6207.974160] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6208.194723] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6212.875248] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6223.329089] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6223.330221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6223.330794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6223.332712] Lustre: Skipped 3 previous similar messages [ 6223.899441] LustreError: 261907:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6223.902046] LustreError: 261907:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6223.938758] Lustre: server umount lustre-MDT0000 complete [ 6225.407097] LustreError: 258883:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326036 with bad export cookie 1667256829627930314 [ 6225.410543] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6225.411479] LustreError: 258883:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6225.531092] Lustre: server umount lustre-MDT0001 complete [ 6238.241519] Lustre: server umount lustre-OST0000 complete [ 6253.641764] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 6279.525897] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 6286.876687] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6287.141263] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6287.996218] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6289.115792] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6295.606319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6295.641436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6295.732207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6295.745764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6295.782666] Lustre: lustre-MDT0000: new disk, initializing [ 6295.815401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6295.817902] Lustre: Skipped 1 previous similar message [ 6295.824895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6297.476458] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6305.640611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6305.665024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6305.707133] Lustre: 271249:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6305.811686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6307.521420] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6311.176563] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6312.788160] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6318.022887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6318.063260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6320.416201] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6321.660384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6321.702473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6323.943512] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6325.530125] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6325.615500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6327.032766] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6327.116372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6331.873755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6331.873807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6331.878659] LustreError: Skipped 1 previous similar message [ 6331.879304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6331.883354] Lustre: Skipped 2 previous similar messages [ 6331.885048] Lustre: Skipped 2 previous similar messages [ 6334.006758] LustreError: 274876:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6334.013633] LustreError: 274876:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6334.083307] Lustre: server umount lustre-OST0000 complete [ 6342.502368] Lustre: server umount lustre-MDT0000 complete [ 6344.433168] LustreError: 271240:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326155 with bad export cookie 1667256829627947128 [ 6344.437345] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6344.437524] LustreError: 271240:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6344.584555] Lustre: server umount lustre-MDT0001 complete [ 6348.374773] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 22:55:59 (1767326159) [ 6352.553058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6352.759817] LustreError: 276240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6354.444355] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6359.513980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6361.231385] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6363.423090] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6364.957827] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6368.822295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6371.228487] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6373.310940] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6382.741279] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 22:56:33 (1767326193) [ 6384.607857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6384.612055] LustreError: Skipped 1 previous similar message [ 6384.614630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6384.618790] Lustre: Skipped 2 previous similar messages [ 6389.914123] Lustre: server umount lustre-OST0000 complete [ 6398.944649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6398.949509] Lustre: Skipped 11 previous similar messages [ 6406.111160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6406.150933] LustreError: 279314:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6406.153613] LustreError: 279314:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 6406.188934] Lustre: server umount lustre-MDT0000 complete [ 6407.961782] LustreError: 276220:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326219 with bad export cookie 1667256829627948010 [ 6407.962478] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6407.968186] LustreError: 276220:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6408.085644] Lustre: server umount lustre-MDT0001 complete [ 6413.204632] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 6414.335953] Key type lgssc unregistered [ 6414.469456] LNet: 280242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6414.472539] LNetError: 280242:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6414.481391] LNet: Removed LNI 192.168.202.117@tcp [ 6414.883387] Key type .llcrypt unregistered [ 6414.884926] Key type ._llcrypt unregistered [ 6445.143799] Key type ._llcrypt registered [ 6445.145211] Key type .llcrypt registered [ 6445.186855] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 6445.503113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6445.510539] alg: No test for adler32 (adler32-zlib) [ 6446.380391] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 6446.482913] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 6448.063137] Key type lgssc registered [ 6448.494121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6451.244663] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6451.251168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6466.719479] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6471.839205] LustreError: 284709:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.117@tcp: failed processing log, type 4: rc = -110 [ 6476.961460] LustreError: 284709:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6476.967096] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6476.969667] LustreError: MGC192.168.202.117@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6476.973071] LustreError: 284709:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6476.976182] LustreError: 284709:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 6476.978857] LustreError: 284709:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 6476.981531] LustreError: 284709:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 6477.006863] Lustre: server umount lustre-OST0000 complete [ 6477.008470] LustreError: 284709:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6482.527409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6483.697021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6485.096968] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6489.510895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6489.655070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6491.172113] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6493.109945] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6494.529168] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6498.467053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6498.544037] LustreError: 287521:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6498.549303] Lustre: 287521:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.202.117@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6498.610165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6500.841153] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6502.901945] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6503.704526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6503.705645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6503.709415] Lustre: Skipped 1 previous similar message [ 6503.712836] Lustre: Skipped 2 previous similar messages [ 6509.026014] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6509.029449] Lustre: Skipped 2 previous similar messages [ 6514.144744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6514.149521] Lustre: Skipped 2 previous similar messages [ 6518.239311] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6518.285640] LustreError: 288173:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6518.288341] LustreError: 288173:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6518.320786] Lustre: server umount lustre-MDT0000 complete [ 6519.267766] LustreError: 285543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6519.274901] LustreError: 285543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6521.827545] LustreError: 285527:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326333 with bad export cookie 3584867777290775206 [ 6521.831658] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6521.835675] LustreError: 285527:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6521.897751] LustreError: 288626:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6521.899874] LustreError: 288626:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6521.959970] Lustre: server umount lustre-MDT0001 complete [ 6536.624518] LustreError: 289219:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6536.627598] LustreError: 289219:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6536.672723] Lustre: server umount lustre-OST0000 complete [ 6542.776756] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 6565.820702] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 6572.027856] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6572.374032] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6572.985524] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6573.810308] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6579.696706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6579.723249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6579.821081] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6579.835702] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6579.873306] Lustre: lustre-MDT0000: new disk, initializing [ 6579.911792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6579.920815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6581.343606] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6589.611501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6589.636153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6589.671050] Lustre: 296613:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6589.696598] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6589.698794] Lustre: Skipped 1 previous similar message [ 6589.736037] Lustre: lustre-MDT0001: new disk, initializing [ 6589.769104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6589.780405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6589.783836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6591.236768] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6594.525736] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6595.716909] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6600.152336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6600.174154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6600.268371] Lustre: lustre-OST0000: new disk, initializing [ 6600.270948] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6600.296479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6602.367737] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6605.786422] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6605.833278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6605.838386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6605.870917] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6607.330053] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6607.414304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6609.023689] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6609.103351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6610.916631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6610.917043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6610.919956] LustreError: Skipped 1 previous similar message [ 6610.925226] Lustre: Skipped 1 previous similar message [ 6610.927620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6610.930933] Lustre: Skipped 1 previous similar message [ 6615.855065] LustreError: 300245:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6615.857393] LustreError: 300245:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6615.900235] Lustre: server umount lustre-OST0000 complete [ 6620.640190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6620.642966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6620.647379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6623.879041] LustreError: 300494:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6623.881830] LustreError: 300494:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6623.921679] Lustre: server umount lustre-MDT0000 complete [ 6625.720296] LustreError: 299053:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326436 with bad export cookie 3584867777290775906 [ 6625.722351] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6625.724782] LustreError: 299053:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6625.892903] Lustre: server umount lustre-MDT0001 complete [ 6629.391720] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 23:00:40 (1767326440) [ 6637.534772] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 6638.658812] Key type lgssc unregistered [ 6638.797437] LNet: 302165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6638.802387] LNetError: 302165:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6638.810554] LNet: Removed LNI 192.168.202.117@tcp [ 6639.149365] Key type .llcrypt unregistered [ 6639.150545] Key type ._llcrypt unregistered [ 6671.970516] Key type ._llcrypt registered [ 6671.971368] Key type .llcrypt registered [ 6672.006209] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 6672.389480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6672.417713] alg: No test for adler32 (adler32-zlib) [ 6673.261311] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 6673.346058] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 6674.927185] Key type lgssc registered [ 6675.312445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6677.889129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6677.896729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6679.059493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6680.373828] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6684.454603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6684.601037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6686.044682] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6688.120415] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6689.688048] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6694.206389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6694.304675] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6694.306764] LustreError: 308739:0:(llog.c:1684:llog_backup()) MGC192.168.202.117@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6694.310970] Lustre: 308739:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.202.117@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6694.356096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6696.403696] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6698.364761] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6699.488529] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6699.489318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6699.493708] Lustre: Skipped 2 previous similar messages [ 6699.495669] Lustre: Skipped 1 previous similar message [ 6704.608269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6704.610091] Lustre: Skipped 3 previous similar messages [ 6705.210733] LustreError: 309391:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6705.235971] Lustre: server umount lustre-MDT0000 complete [ 6707.934403] LustreError: 306698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326519 with bad export cookie 1395751099444022040 [ 6707.938271] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6707.939231] LustreError: 306698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6707.943632] LustreError: 308682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6707.945202] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6707.949216] LustreError: 308682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6707.957639] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6708.008557] LustreError: 309841:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6708.010680] LustreError: 309841:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6708.057574] Lustre: server umount lustre-MDT0001 complete [ 6711.806270] LustreError: 310434:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6711.808231] LustreError: 310434:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6711.844741] Lustre: server umount lustre-OST0000 complete [ 6717.821954] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 6738.552912] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 6744.639465] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6744.879503] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6745.496906] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6746.255900] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6751.491499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6751.513195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6751.589543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6751.602136] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6751.635993] Lustre: lustre-MDT0000: new disk, initializing [ 6751.659980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6751.667547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6752.922613] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6758.905481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6758.928336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6758.958287] Lustre: 317828:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6758.976077] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6758.977427] Lustre: Skipped 1 previous similar message [ 6758.978781] LustreError: 318939:0:(llog.c:1684:llog_backup()) MGC192.168.202.117@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6758.981657] LustreError: 318939:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 6758.983843] Lustre: 318939:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.202.117@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6758.991742] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6758.993623] Lustre: Skipped 1 previous similar message [ 6759.024554] Lustre: lustre-MDT0001: new disk, initializing [ 6759.055256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6759.067223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6759.071270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6760.287944] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6763.195867] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6764.292906] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6768.466351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6768.487580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6768.567254] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6768.568352] Lustre: Skipped 1 previous similar message [ 6768.570080] LustreError: 320243:0:(llog.c:1684:llog_backup()) MGC192.168.202.117@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6768.573173] LustreError: 320243:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 6768.576035] Lustre: 320243:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.202.117@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6768.580383] Lustre: 320243:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6768.588353] Lustre: lustre-OST0000: new disk, initializing [ 6768.590586] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6768.618197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6770.382828] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6770.804040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6770.807701] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6770.839933] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6773.503244] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6774.917288] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6774.991383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6776.240877] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6776.312051] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6781.409651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6781.410409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6781.412192] LustreError: Skipped 1 previous similar message [ 6781.417565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6781.420560] Lustre: Skipped 1 previous similar message [ 6783.019770] LustreError: 321457:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6783.021386] LustreError: 321457:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6783.053380] Lustre: server umount lustre-OST0000 complete [ 6784.991879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6784.995435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6784.999688] Lustre: Skipped 1 previous similar message [ 6786.528184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6786.530098] Lustre: Skipped 2 previous similar messages [ 6790.651240] LustreError: 321706:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6790.653111] LustreError: 321706:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6790.680956] Lustre: server umount lustre-MDT0000 complete [ 6791.647878] LustreError: 317839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6791.653807] LustreError: 317839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6792.185098] LustreError: 317819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326603 with bad export cookie 1395751099444022733 [ 6792.187967] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6792.189259] LustreError: 317819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6792.298093] Lustre: server umount lustre-MDT0001 complete [ 6795.081383] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 23:03:25 (1767326605) [ 6818.727452] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 6824.226213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6824.381533] LustreError: 326736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6824.396732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6825.730112] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6827.564754] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6829.535782] LustreError: 326736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6831.007797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6832.460911] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6834.233395] Lustre: 328366:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6837.495766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6839.498522] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6844.273307] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6844.298028] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6844.356287] Lustre: lustre-OST0001: new disk, initializing [ 6844.358632] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6846.234964] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6848.447989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6848.451210] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6848.482235] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6856.144162] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6860.686078] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6866.270363] Lustre: DEBUG MARKER: ADD OST3 [ 6867.887314] loop0: detected capacity change from 0 to 204800000 [ 6868.012261] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6868.019707] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6868.025181] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6868.077759] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6871.989647] loop0: detected capacity change from 0 to 204800000 [ 6873.004674] Lustre: DEBUG MARKER: STOP OST3 [ 6873.882578] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6873.906794] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6873.972075] Lustre: lustre-OST0003: new disk, initializing [ 6873.973991] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 6874.024180] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 6874.027386] Lustre: Skipped 3 previous similar messages [ 6875.110775] Lustre: Failing over lustre-OST0003 [ 6875.135075] LustreError: 332532:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 6875.136904] LustreError: 332532:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6875.154814] Lustre: server umount lustre-OST0003 complete [ 6875.710787] LustreError: 328913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6875.984165] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6879.462444] Lustre: DEBUG MARKER: START OST3 again [ 6883.335549] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6885.281649] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6885.371847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 6885.376262] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 6885.386916] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 6898.876684] Lustre: Failing over lustre-OST0003 [ 6898.899939] LustreError: 334818:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 6898.901760] LustreError: 334818:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6898.913062] Lustre: server umount lustre-OST0003 complete [ 6900.704506] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6900.705239] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6900.706431] LustreError: Skipped 1 previous similar message [ 6900.709418] Lustre: Skipped 1 previous similar message [ 6900.711123] LustreError: 328913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6900.716626] LustreError: 328913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6912.992040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6912.994032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6912.999081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6917.533236] Lustre: server umount lustre-MDT0000 complete [ 6919.141112] LustreError: 330078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6919.145592] LustreError: 330078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6920.323537] LustreError: 330824:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326731 with bad export cookie 1395751099444023615 [ 6920.326530] LustreError: 330824:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6920.328298] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6926.586790] Lustre: server umount lustre-MDT0001 complete [ 6930.445463] Lustre: server umount lustre-OST0000 complete [ 6933.301122] LustreError: 336583:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6933.304708] LustreError: 336583:0:(obd_class.h:479:obd_check_dev()) Skipped 27 previous similar messages [ 6933.358424] Lustre: server umount lustre-OST0001 complete [ 6937.895676] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 6944.034024] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 6950.867968] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6950.892783] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6951.370677] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6952.110981] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6957.490802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6957.516696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6957.612608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6957.626680] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6957.670663] Lustre: lustre-MDT0000: new disk, initializing [ 6957.697702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6957.699801] Lustre: Skipped 1 previous similar message [ 6957.705907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6959.076497] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6965.592320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6965.621637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6965.658619] Lustre: 341467:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6965.788906] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6967.271902] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6970.465396] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6971.707790] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6975.777940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6975.798283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6975.895344] Lustre: lustre-OST0000: new disk, initializing [ 6975.897194] Lustre: Skipped 1 previous similar message [ 6975.899557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6975.902090] Lustre: Skipped 2 previous similar messages [ 6977.704334] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6977.913497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6977.915773] Lustre: Skipped 1 previous similar message [ 6977.917351] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6977.953860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6980.798046] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6982.139759] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6982.218480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6983.510214] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6983.585372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6988.258677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6988.259194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6988.260000] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6988.260025] Lustre: Skipped 8 previous similar messages [ 6988.261298] LustreError: Skipped 1 previous similar message [ 6988.269259] Lustre: Skipped 6 previous similar messages [ 6990.352860] Lustre: server umount lustre-OST0000 complete [ 6998.084804] LustreError: 345343:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6998.087488] LustreError: 345343:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6998.125893] Lustre: server umount lustre-MDT0000 complete [ 6998.496401] LustreError: 341477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6998.501936] LustreError: 341477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6999.724464] LustreError: 342593:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326810 with bad export cookie 1395751099444025876 [ 6999.728666] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6999.729386] LustreError: 342593:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6999.857208] Lustre: server umount lustre-MDT0001 complete [ 7001.044793] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 23:06:51 (1767326811) [ 7012.635774] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 7031.493525] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 7037.744662] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7037.839504] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7038.352456] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7039.175023] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7044.037796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7044.058964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7044.140147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7044.150814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7044.184608] Lustre: lustre-MDT0000: new disk, initializing [ 7044.213116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7045.408482] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7051.158919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7051.175938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7051.201916] Lustre: 354137:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7051.287437] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7052.591332] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7055.702927] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7056.955580] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7061.432716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7061.457231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7063.144070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7063.560403] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7066.484952] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7067.764628] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7067.834372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7069.088169] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7069.156728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7073.248125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7073.250368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7073.253853] Lustre: Skipped 2 previous similar messages [ 7073.255691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7073.257494] Lustre: Skipped 3 previous similar messages [ 7075.911068] Lustre: server umount lustre-OST0000 complete [ 7083.423590] Lustre: server umount lustre-MDT0000 complete [ 7084.000101] LustreError: 354146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7084.006056] LustreError: 354146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7085.035492] LustreError: 354127:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326896 with bad export cookie 1395751099444026765 [ 7085.039316] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7085.041924] LustreError: 354127:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7085.179220] Lustre: server umount lustre-MDT0001 complete [ 7089.348480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7089.505413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7089.507817] Lustre: Skipped 5 previous similar messages [ 7090.719477] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7094.450186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7095.902287] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7097.724352] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7098.950638] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7102.398348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7104.492899] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7106.356184] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7124.961913] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7124.965843] LustreError: Skipped 2 previous similar messages [ 7129.261100] LustreError: 361678:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7129.263940] LustreError: 361678:0:(obd_class.h:479:obd_check_dev()) Skipped 31 previous similar messages [ 7129.307399] Lustre: server umount lustre-OST0000 complete [ 7137.113799] Lustre: server umount lustre-MDT0000 complete [ 7138.880409] LustreError: 361929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326950 with bad export cookie 1395751099444027647 [ 7138.884323] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7138.886712] LustreError: 361929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7139.020635] Lustre: server umount lustre-MDT0001 complete [ 7237.718560] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 7238.678224] Key type lgssc unregistered [ 7238.818360] LNet: 362887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7238.820389] LNetError: 362887:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7238.827298] LNet: Removed LNI 192.168.202.117@tcp [ 7239.108121] Key type .llcrypt unregistered [ 7239.109587] Key type ._llcrypt unregistered [ 7253.298363] Key type ._llcrypt registered [ 7253.300384] Key type .llcrypt registered [ 7253.340301] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 7276.937965] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 7277.375762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7277.399232] alg: No test for adler32 (adler32-zlib) [ 7278.268312] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 7278.371443] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 7279.959159] Key type lgssc registered [ 7280.391315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7286.078369] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7286.394481] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7287.054355] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7287.807873] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7292.963861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7292.987600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7292.991509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7294.073830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7294.085399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7294.126650] Lustre: lustre-MDT0000: new disk, initializing [ 7294.152767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7294.159245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7295.521640] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7301.880658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7301.911219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7301.948907] Lustre: 371887:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7301.985568] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7301.989196] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7301.991337] Lustre: Skipped 1 previous similar message [ 7302.028647] Lustre: lustre-MDT0001: new disk, initializing [ 7302.063304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7302.075668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7302.079869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7303.420356] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7306.514640] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7307.780818] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7312.281494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7312.303163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7312.401719] Lustre: lustre-OST0000: new disk, initializing [ 7312.403554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7312.432046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7314.358684] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7317.547451] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7318.944286] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7320.057736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7320.062518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7320.091955] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7321.054869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 7322.473916] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7322.567324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7325.155626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7325.155793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7325.156452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7325.157793] LustreError: Skipped 1 previous similar message [ 7325.168017] Lustre: Skipped 1 previous similar message [ 7329.327115] LustreError: 375531:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7329.369239] Lustre: server umount lustre-OST0000 complete [ 7332.832053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7332.834346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7332.838231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7332.841057] Lustre: Skipped 1 previous similar message [ 7335.392184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7335.392908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7337.088697] LustreError: 375780:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7337.091958] LustreError: 375780:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7337.133126] Lustre: server umount lustre-MDT0000 complete [ 7338.727150] LustreError: 371877:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327149 with bad export cookie 10825927045574079074 [ 7338.730337] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7338.731925] LustreError: 371877:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7338.803715] LustreError: 376030:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7338.805516] LustreError: 376030:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7338.852092] Lustre: server umount lustre-MDT0001 complete [ 7363.952162] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 7369.502754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7369.677114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7371.059218] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7372.984622] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7374.817150] LustreError: 380566:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7374.821538] LustreError: 380566:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7376.497181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7376.615854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7377.973743] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7379.808160] Lustre: 382196:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7383.090314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7385.172795] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7390.307097] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7390.327079] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7390.381740] Lustre: lustre-OST0001: new disk, initializing [ 7390.383638] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7390.415797] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7390.417739] Lustre: Skipped 1 previous similar message [ 7392.295881] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7395.330889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7395.333863] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7395.367355] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7401.908811] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7405.513959] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7414.703060] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 23:13:45 (1767327225) [ 7415.778609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7415.781648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7415.783148] Lustre: Skipped 3 previous similar messages [ 7415.786177] Lustre: Skipped 3 previous similar messages [ 7421.504378] LustreError: 385015:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7421.506394] LustreError: 385015:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7421.541571] Lustre: server umount lustre-MDT0000 complete [ 7424.250164] LustreError: 382198:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327235 with bad export cookie 10825927045574079956 [ 7424.253940] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7424.256229] LustreError: 382198:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7424.372666] Lustre: server umount lustre-MDT0001 complete [ 7438.387093] LustreError: 386060:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7438.389334] LustreError: 386060:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 7438.406899] Lustre: server umount lustre-OST0000 complete [ 7451.636389] LustreError: 386511:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7451.639405] LustreError: 386511:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7451.695979] Lustre: server umount lustre-OST0001 complete [ 7456.016660] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 7475.683081] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 7481.859731] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7482.078446] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7482.770058] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7483.518554] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7488.574684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7488.595697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7488.676491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7488.687740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7488.724704] Lustre: lustre-MDT0000: new disk, initializing [ 7488.747800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7488.755035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7490.020138] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7496.236256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7496.256198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7496.287928] Lustre: 393707:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7496.393483] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7497.674406] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7500.731199] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7501.999758] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7506.311839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7506.332513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7506.433717] Lustre: lustre-OST0000: new disk, initializing [ 7506.434903] Lustre: Skipped 1 previous similar message [ 7506.436502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7506.438107] Lustre: Skipped 2 previous similar messages [ 7508.023337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7508.026821] Lustre: Skipped 1 previous similar message [ 7508.030758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7508.067466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7508.303434] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7511.414365] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7512.778254] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7512.849791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7514.152785] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7514.229585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7518.179544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7518.179843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7518.186048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7518.186658] LustreError: Skipped 1 previous similar message [ 7518.190157] Lustre: Skipped 6 previous similar messages [ 7520.942082] LustreError: 397333:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7520.944041] LustreError: 397333:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7520.983742] Lustre: server umount lustre-OST0000 complete [ 7528.738720] Lustre: server umount lustre-MDT0000 complete [ 7530.294480] LustreError: 394832:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327341 with bad export cookie 10825927045574082119 [ 7530.299928] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7530.429260] Lustre: server umount lustre-MDT0001 complete [ 7534.537565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7534.698433] LustreError: 398502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7534.721850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7534.724983] Lustre: Skipped 2 previous similar messages [ 7536.060426] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7540.132049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7540.191765] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7540.193462] Lustre: Skipped 4 previous similar messages [ 7541.665613] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7543.448955] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7544.661982] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7548.048426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7549.920264] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7551.569470] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7697.891612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7697.893069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7697.894895] Lustre: Skipped 4 previous similar messages [ 7697.898424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7700.333089] LustreError: 401174:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7700.335216] LustreError: 401174:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7700.370501] Lustre: server umount lustre-OST0000 complete [ 7716.319097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7716.371089] Lustre: server umount lustre-MDT0000 complete [ 7717.879024] LustreError: 398527:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327529 with bad export cookie 10825927045574083001 [ 7717.882099] LustreError: 398527:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7717.882398] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7717.990484] Lustre: server umount lustre-MDT0001 complete [ 7815.622905] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 7816.632935] Key type lgssc unregistered [ 7816.747343] LNet: 402386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7816.749559] LNetError: 402386:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7816.758294] LNet: Removed LNI 192.168.202.117@tcp [ 7817.049103] Key type .llcrypt unregistered [ 7817.050125] Key type ._llcrypt unregistered [ 7831.578354] Key type ._llcrypt registered [ 7831.579425] Key type .llcrypt registered [ 7831.620299] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 7855.192959] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 7855.557247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7855.607875] alg: No test for adler32 (adler32-zlib) [ 7856.467873] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 7856.567026] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 7858.143153] Key type lgssc registered [ 7858.523450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7864.512204] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7864.665542] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7865.174975] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7865.920574] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7889.462861] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 7895.653235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7895.671819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7895.676285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7896.760340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7896.774573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7896.811623] Lustre: lustre-MDT0000: new disk, initializing [ 7896.834445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7896.840635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7898.090028] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7901.189675] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7905.434750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7905.460690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7905.493440] Lustre: 415159:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7905.518070] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7905.520679] Lustre: Skipped 1 previous similar message [ 7905.555158] Lustre: lustre-MDT0001: new disk, initializing [ 7905.589439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7905.603070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7905.606482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7906.878583] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7910.030434] Lustre: 417075:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7914.449723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7914.479708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7914.579337] Lustre: lustre-OST0000: new disk, initializing [ 7914.581949] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7914.614591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7915.448921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7915.452296] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7915.501204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7916.546949] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7922.771393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7922.796186] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7922.861462] Lustre: lustre-OST0001: new disk, initializing [ 7922.863470] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7922.901197] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7924.838623] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7928.328796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7928.333050] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7928.364567] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7930.328989] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7937.108730] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7946.720090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7946.722667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7946.726874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7948.771308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7948.772832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7948.775248] Lustre: Skipped 1 previous similar message [ 7948.778333] Lustre: Skipped 1 previous similar message [ 7951.050995] LustreError: 420053:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7951.083664] Lustre: server umount lustre-MDT0000 complete [ 7952.263600] LustreError: 416477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327763 with bad export cookie 10736425569228470491 [ 7952.267103] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7952.267495] LustreError: 416477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7952.342053] LustreError: 420254:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7952.343615] LustreError: 420254:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7952.394848] Lustre: server umount lustre-MDT0001 complete [ 7964.727794] LustreError: 420599:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7964.730440] LustreError: 420599:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7964.750259] Lustre: server umount lustre-OST0000 complete [ 7976.247991] LustreError: 420800:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7976.250487] LustreError: 420800:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7976.303531] Lustre: server umount lustre-OST0001 complete [ 7981.905873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7987.340818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7987.667559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7990.608346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7995.039209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7995.355698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8005.675044] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 8011.064946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8011.072235] Lustre: lustre-MDT0000: reset Object Index mappings [ 8011.249172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8012.547516] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8014.347473] Lustre: 426684:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8016.351852] LustreError: 426130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8016.360590] LustreError: 426130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8017.776155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8017.781550] Lustre: lustre-MDT0001: reset Object Index mappings [ 8019.315644] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8021.147901] Lustre: 427773:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8024.465993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8024.601748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8024.604674] Lustre: Skipped 1 previous similar message [ 8026.453793] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8028.708222] LustreError: 428321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8030.341166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8032.283348] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8035.940182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8035.940225] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8040.342395] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8041.517057] Lustre: 430002:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8048.607923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8048.610345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8048.613992] Lustre: Skipped 1 previous similar message [ 8048.615744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8051.682069] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8051.683717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8051.687024] Lustre: Skipped 2 previous similar messages [ 8051.690031] Lustre: Skipped 2 previous similar messages [ 8053.828816] LustreError: 430108:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8053.832282] LustreError: 430108:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8053.873563] Lustre: server umount lustre-MDT0000 complete [ 8055.144660] LustreError: 426110:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327866 with bad export cookie 10736425569228477302 [ 8055.147341] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8055.150312] LustreError: 426110:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8055.278986] Lustre: server umount lustre-MDT0001 complete [ 8067.699084] LustreError: 430654:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 8067.700821] LustreError: 430654:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 8067.719588] Lustre: server umount lustre-OST0000 complete [ 8079.194338] Lustre: server umount lustre-OST0001 complete [ 8086.323933] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 23:24:57 (1767327897) [ 8099.498373] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 8120.436993] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 8127.574735] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8127.830647] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8128.428559] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8129.242676] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8134.426133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8134.446583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8134.526190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8134.536284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8134.572707] Lustre: lustre-MDT0000: new disk, initializing [ 8134.595302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8134.597019] Lustre: Skipped 1 previous similar message [ 8134.602349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8135.858223] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8142.095292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8142.122241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8142.163324] Lustre: 440418:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8142.264910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8143.641121] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8146.720396] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8147.902522] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8152.378776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8152.401027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8152.513331] Lustre: lustre-OST0000: new disk, initializing [ 8152.514823] Lustre: Skipped 1 previous similar message [ 8152.517270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8152.519442] Lustre: Skipped 2 previous similar messages [ 8154.372033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8154.374803] Lustre: Skipped 1 previous similar message [ 8154.377174] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8154.430523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8154.522373] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8157.676426] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8159.095831] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8159.183902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8160.517893] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8160.584698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8164.834539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8164.835515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8164.841799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8164.845016] Lustre: Skipped 2 previous similar messages [ 8167.277106] LustreError: 444041:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8167.280069] LustreError: 444041:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 8167.318101] Lustre: server umount lustre-OST0000 complete [ 8175.072252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8175.074213] Lustre: Skipped 3 previous similar messages [ 8183.263173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8183.332624] Lustre: server umount lustre-MDT0000 complete [ 8184.923420] LustreError: 440408:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327996 with bad export cookie 10736425569228479416 [ 8184.926992] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8184.928675] LustreError: 440408:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8185.062943] Lustre: server umount lustre-MDT0001 complete [ 8209.556799] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 8214.937588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8215.083563] LustreError: 449076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8215.099470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8215.101211] Lustre: Skipped 2 previous similar messages [ 8216.411859] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8218.258820] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8220.129075] LustreError: 449076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8221.556578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8223.061228] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8224.863244] Lustre: 450707:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8228.311887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8230.221698] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8235.178273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8235.201121] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8235.263457] Lustre: lustre-OST0001: new disk, initializing [ 8235.265364] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8236.363706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8236.368102] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8236.413536] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8237.155618] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8241.508234] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8242.681874] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8249.495657] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8249.502426] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8252.384580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8252.387319] LustreError: Skipped 1 previous similar message [ 8252.388669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8252.392503] Lustre: Skipped 3 previous similar messages [ 8252.394398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8252.396461] Lustre: Skipped 2 previous similar messages [ 8257.342618] LustreError: 453655:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8257.344565] LustreError: 453655:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8257.375267] Lustre: server umount lustre-MDT0000 complete [ 8258.629449] LustreError: 449057:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767328069 with bad export cookie 10736425569228480298 [ 8258.633364] LustreError: 449057:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8258.635271] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8258.764679] Lustre: server umount lustre-MDT0001 complete [ 8271.365483] Lustre: server umount lustre-OST0000 complete [ 8281.898925] Lustre: server umount lustre-OST0001 complete [ 8292.217934] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 8297.543917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8297.699763] LustreError: 456780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8297.719412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8297.722306] Lustre: Skipped 3 previous similar messages [ 8299.031302] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8300.833747] Lustre: 457332:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8304.267223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8305.733448] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8311.051398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8313.130652] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8314.276310] LustreError: 458969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8314.280951] LustreError: 458969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8317.263230] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8319.234433] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8322.158283] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 8322.182668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 8327.866440] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8336.029283] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8336.038283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8339.044954] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 23:29:09 (1767328149) [ 8350.687919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8350.691240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8350.695064] Lustre: Skipped 3 previous similar messages [ 8350.696880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8350.699007] Lustre: Skipped 4 previous similar messages [ 8356.677836] LustreError: 461367:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8356.680615] LustreError: 461367:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 8356.710952] Lustre: server umount lustre-MDT0000 complete [ 8358.369479] LustreError: 456779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8358.375821] LustreError: 456779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8359.667860] LustreError: 456761:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767328170 with bad export cookie 10736425569228486605 [ 8359.672891] LustreError: 456761:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8359.674763] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8365.901063] Lustre: server umount lustre-MDT0001 complete [ 8384.991150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8385.033931] Lustre: server umount lustre-OST0000 complete [ 8393.946287] Lustre: server umount lustre-OST0001 complete [ 8398.180226] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 8418.461704] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 8425.253464] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8425.681848] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8426.285816] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8427.354485] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8452.021729] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 8458.481784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8458.502234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8458.581828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8458.594493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8458.632993] Lustre: lustre-MDT0000: new disk, initializing [ 8458.655384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8458.657152] Lustre: Skipped 3 previous similar messages [ 8458.662862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8459.998915] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8467.722071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8467.751992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8467.797503] Lustre: 473780:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8467.803840] Lustre: 473780:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 8467.931063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8469.357692] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8477.407687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8477.431313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8478.682830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8479.375378] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8485.790717] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8485.821872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8487.285895] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8487.796135] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8492.376296] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8493.557158] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8493.559402] Lustre: Skipped 1 previous similar message [ 8502.997976] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 23:31:53 (1767328313) [ 8503.775822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8503.777851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8503.780979] Lustre: Skipped 5 previous similar messages [ 8503.782920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8503.784329] Lustre: Skipped 8 previous similar messages [ 8509.787404] LustreError: 478762:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8509.791117] LustreError: 478762:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 8509.830900] Lustre: server umount lustre-MDT0000 complete [ 8511.375528] LustreError: 473773:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767328322 with bad export cookie 10736425569228494480 [ 8511.381482] LustreError: 473773:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8511.381501] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8511.543967] Lustre: server umount lustre-MDT0001 complete [ 8524.108018] Lustre: server umount lustre-OST0000 complete [ 8535.832293] Lustre: server umount lustre-OST0001 complete [ 8549.678776] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 8569.331273] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 8575.474630] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8575.819844] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8576.354778] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8577.048551] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8582.443428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8582.467172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8582.556185] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8582.570019] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8582.571651] Lustre: Skipped 4 previous similar messages [ 8582.606609] Lustre: lustre-MDT0000: new disk, initializing [ 8582.607792] Lustre: Skipped 3 previous similar messages [ 8582.636310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8582.641052] Lustre: Skipped 3 previous similar messages [ 8583.856445] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8589.926220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8589.947573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8590.097172] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8590.099624] Lustre: Skipped 2 previous similar messages [ 8591.436469] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8594.486035] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8595.640942] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8600.160967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8600.192626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8602.236397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8602.270168] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8605.267045] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8605.939497] Lustre: Mounted lustre-client [ 8606.251455] LustreError: 491591:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff92b606857000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 8606.278160] Lustre: Unmounted lustre-client [ 8611.281782] Lustre: server umount lustre-OST0000 complete [ 8615.904125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8615.906237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8615.909718] Lustre: Skipped 3 previous similar messages [ 8627.167107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8627.247415] Lustre: server umount lustre-MDT0000 complete [ 8627.679117] Lustre: 407917:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767328422/real 1767328422] req@ffff92b707905180 x1853177376100736/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1767328438 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8627.687762] LustreError: 488409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8627.692123] LustreError: 488409:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8628.697593] LustreError: 490833:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767328439 with bad export cookie 10736425569228496027 [ 8628.701298] Lustre: 407915:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767328423/real 1767328423] req@ffff92b709ed1f80 x1853177376100992/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1767328439 ref 1 fl Rpc:EXQr/200/ffffffff rc -5/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 8628.702715] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8628.826633] Lustre: server umount lustre-MDT0001 complete [ 8633.046266] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 8634.138358] Key type lgssc unregistered [ 8665.567206] LNet: 1 local NIs in recovery (showing 1): 192.168.202.117@tcp [ 8666.015325] LNet: 493107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8666.017557] LNetError: 493107:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8738.725310] LNet: Removed LNI 192.168.202.117@tcp [ 8739.054125] Key type .llcrypt unregistered [ 8739.055154] Key type ._llcrypt unregistered [ 8751.551449] Key type ._llcrypt registered [ 8751.552311] Key type .llcrypt registered [ 8751.591196] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 8774.931671] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 8775.239500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8775.275415] alg: No test for adler32 (adler32-zlib) [ 8776.120952] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 8776.218638] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 8777.799166] Key type lgssc registered [ 8778.166479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8783.779359] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8783.999257] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8784.518181] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8785.356314] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8785.910913] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 23:36:36 (1767328596) [ 8790.014598] Lustre: DEBUG MARKER: conf-sanity test_161: @@@@@@ FAIL: MDT start failed [ 8791.804597] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 23:36:42 (1767328602) [ 8824.580851] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 23:37:15 (1767328635) [ 8854.766649] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 8855.875367] Key type lgssc unregistered [ 8856.021430] LNet: 510300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8856.023737] LNetError: 510300:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8856.032431] LNet: Removed LNI 192.168.202.117@tcp [ 8856.362351] Key type .llcrypt unregistered [ 8856.363470] Key type ._llcrypt unregistered [ 8913.861492] Key type ._llcrypt registered [ 8913.862414] Key type .llcrypt registered [ 8913.901830] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 8914.007292] Key type .llcrypt unregistered [ 8914.008579] Key type ._llcrypt unregistered [ 8938.272791] Key type ._llcrypt registered [ 8938.273770] Key type .llcrypt registered [ 8938.313307] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 8938.405366] Key type .llcrypt unregistered [ 8938.406566] Key type ._llcrypt unregistered [ 8963.006768] Key type ._llcrypt registered [ 8963.007720] Key type .llcrypt registered [ 8963.050383] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 8963.159430] Key type .llcrypt unregistered [ 8963.161310] Key type ._llcrypt unregistered [ 9018.687565] Key type ._llcrypt registered [ 9018.688444] Key type .llcrypt registered [ 9018.722221] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 9018.807431] Key type .llcrypt unregistered [ 9018.808901] Key type ._llcrypt unregistered [ 9043.168752] Key type ._llcrypt registered [ 9043.170023] Key type .llcrypt registered [ 9043.202838] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 9043.290201] Key type .llcrypt unregistered [ 9043.291147] Key type ._llcrypt unregistered [ 9067.196293] Key type ._llcrypt registered [ 9067.197417] Key type .llcrypt registered [ 9067.233522] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 9067.331339] Key type .llcrypt unregistered [ 9067.332386] Key type ._llcrypt unregistered [ 9092.377959] Key type ._llcrypt registered [ 9092.380333] Key type .llcrypt registered [ 9092.422821] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 9092.523435] Key type .llcrypt unregistered [ 9092.525056] Key type ._llcrypt unregistered [ 9117.955586] Key type ._llcrypt registered [ 9117.956618] Key type .llcrypt registered [ 9117.990100] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 9118.084364] Key type .llcrypt unregistered [ 9118.085983] Key type ._llcrypt unregistered [ 9143.357774] Key type ._llcrypt registered [ 9143.359243] Key type .llcrypt registered [ 9143.393697] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 9143.503683] Key type .llcrypt unregistered [ 9143.504609] Key type ._llcrypt unregistered [ 9180.381908] Key type ._llcrypt registered [ 9180.382926] Key type .llcrypt registered [ 9180.416534] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 9180.505367] Key type .llcrypt unregistered [ 9180.506926] Key type ._llcrypt unregistered [ 9205.209871] Key type ._llcrypt registered [ 9205.211320] Key type .llcrypt registered [ 9205.255080] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 9205.355171] Key type .llcrypt unregistered [ 9205.356208] Key type ._llcrypt unregistered [ 9230.229404] Key type ._llcrypt registered [ 9230.230489] Key type .llcrypt registered [ 9230.273720] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 9230.373422] Key type .llcrypt unregistered [ 9230.375179] Key type ._llcrypt unregistered [ 9266.902864] Key type ._llcrypt registered [ 9266.903815] Key type .llcrypt registered [ 9266.944889] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 9290.226934] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 9290.606873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9290.621164] alg: No test for adler32 (adler32-zlib) [ 9291.481204] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 9291.571923] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 9293.159179] Key type lgssc registered [ 9293.531728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9299.762382] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9300.015478] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9300.648802] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9301.381332] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9301.918635] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8906 sec ========== 23:45:12 (1767329112) [ 9302.491721] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:45:13 (1767329113) === [ 9303.476136] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:45:14 (1767329114) === [ 9312.440428] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 9313.319644] Key type lgssc unregistered [ 9313.439341] LNet: 570652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9313.441311] LNetError: 570652:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9313.448304] LNet: Removed LNI 192.168.202.117@tcp [ 9313.702402] Key type .llcrypt unregistered [ 9313.703343] Key type ._llcrypt unregistered