[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478167773 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002358] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007677] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010129] LSM: Security Framework initializing [ 0.011050] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.013067] *** VALIDATE selinux *** [ 0.021150] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026246] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028103] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029125] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030086] *** VALIDATE tmpfs *** [ 0.032223] *** VALIDATE proc *** [ 0.033220] *** VALIDATE cgroup *** [ 0.034007] *** VALIDATE cgroup2 *** [ 0.036020] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037115] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039029] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.043024] debug: unmapping init [mem 0xffffffff92459000-0xffffffff92460fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046671] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047018] ... version: 2 [ 0.048010] ... bit width: 48 [ 0.049011] ... generic registers: 4 [ 0.049980] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051008] ... fixed-purpose events: 3 [ 0.051890] ... event mask: 000000070000000f [ 0.052272] rcu: Hierarchical SRCU implementation. [ 0.054325] smp: Bringing up secondary CPUs ... [ 0.055586] x86: Booting SMP configuration: [ 0.056020] .... node #0, CPUs: #1 #2 #3 [ 0.059145] smp: Brought up 1 node, 4 CPUs [ 0.061013] smpboot: Max logical packages: 1 [ 0.062018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142141] node 0 deferred pages initialised in 77ms [ 0.145150] devtmpfs: initialized [ 0.146259] x86/mm: Memory block size: 128MB [ 0.148911] gcov: version magic: 0x41383552 [ 0.150321] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152265] pinctrl core: initialized pinctrl subsystem [ 0.153210] [ 0.153928] ************************************************************* [ 0.154015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155011] ** ** [ 0.156016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157013] ** ** [ 0.158014] ** This means that this kernel is built to expose internal ** [ 0.159014] ** IOMMU data structures, which may compromise security on ** [ 0.160013] ** your system. ** [ 0.161015] ** ** [ 0.162014] ** If you see this message and you are not debugging the ** [ 0.163015] ** kernel, report this immediately to your vendor! ** [ 0.164014] ** ** [ 0.165014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166013] ************************************************************* [ 0.167703] NET: Registered protocol family 16 [ 0.168446] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.170065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.171453] cpuidle: using governor menu [ 0.173355] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.174539] PCI: Using configuration type 1 for base access [ 0.178123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.187106] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.188026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.190031] cryptd: max_cpu_qlen set to 1000 [ 0.192109] ACPI: Added _OSI(Module Device) [ 0.193012] ACPI: Added _OSI(Processor Device) [ 0.194012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.195010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200283] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.207388] ACPI: Interpreter enabled [ 0.208047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.209009] ACPI: Using IOAPIC for interrupt routing [ 0.210077] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.212360] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.221616] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.223028] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.224009] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.226057] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.229153] acpiphp: Slot [2] registered [ 0.230100] acpiphp: Slot [5] registered [ 0.231066] acpiphp: Slot [6] registered [ 0.231988] acpiphp: Slot [7] registered [ 0.234141] acpiphp: Slot [8] registered [ 0.235106] acpiphp: Slot [9] registered [ 0.236089] acpiphp: Slot [10] registered [ 0.238092] acpiphp: Slot [3] registered [ 0.239080] acpiphp: Slot [4] registered [ 0.241076] acpiphp: Slot [11] registered [ 0.242120] acpiphp: Slot [12] registered [ 0.243087] acpiphp: Slot [13] registered [ 0.245103] acpiphp: Slot [14] registered [ 0.247139] acpiphp: Slot [15] registered [ 0.248092] acpiphp: Slot [16] registered [ 0.250111] acpiphp: Slot [17] registered [ 0.252103] acpiphp: Slot [18] registered [ 0.253108] acpiphp: Slot [19] registered [ 0.254083] acpiphp: Slot [20] registered [ 0.256082] acpiphp: Slot [21] registered [ 0.257090] acpiphp: Slot [22] registered [ 0.259083] acpiphp: Slot [23] registered [ 0.260056] acpiphp: Slot [24] registered [ 0.261107] acpiphp: Slot [25] registered [ 0.262085] acpiphp: Slot [26] registered [ 0.264117] acpiphp: Slot [27] registered [ 0.265075] acpiphp: Slot [28] registered [ 0.267081] acpiphp: Slot [29] registered [ 0.268097] acpiphp: Slot [30] registered [ 0.269091] acpiphp: Slot [31] registered [ 0.271047] PCI host bridge to bus 0000:00 [ 0.273024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.275020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.277022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.280025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.282022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.285021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.286268] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.289127] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.292318] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.302888] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.307979] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.310015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.312012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.314020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.316954] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.320006] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.323042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.325792] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.329976] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.340015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.344013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.349639] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.356015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.363015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.386022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.395069] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.402016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.409014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.424061] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.440436] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.445012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.453015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.470016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.479000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.484013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.494014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.513016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.528000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.535014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.543014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.559017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.570232] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.575012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.579010] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.593017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.606200] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.609379] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.611352] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.613300] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.615190] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.619044] iommu: Default domain type: Passthrough [ 0.621590] SCSI subsystem initialized [ 0.623399] ACPI: bus type USB registered [ 0.625267] usbcore: registered new interface driver usbfs [ 0.628110] usbcore: registered new interface driver hub [ 0.631080] usbcore: registered new device driver usb [ 0.634145] pps_core: LinuxPPS API ver. 1 registered [ 0.637011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.641058] PTP clock support registered [ 0.643074] EDAC MC: Ver: 3.0.0 [ 0.645145] PCI: Using ACPI for IRQ routing [ 0.646000] NetLabel: Initializing [ 0.647011] NetLabel: domain hash size = 128 [ 0.648009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.650110] NetLabel: unlabeled traffic allowed by default [ 0.653113] vgaarb: loaded [ 0.654262] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.655012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.660469] clocksource: Switched to clocksource kvm-clock [ 0.767423] VFS: Disk quotas dquot_6.6.0 [ 0.769406] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.771794] *** VALIDATE ramfs *** [ 0.772951] *** VALIDATE hugetlbfs *** [ 0.774094] pnp: PnP ACPI init [ 0.777149] pnp: PnP ACPI: found 6 devices [ 0.803329] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.806917] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.809146] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.811234] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.813646] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.816025] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.818958] NET: Registered protocol family 2 [ 0.820894] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.825494] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.828618] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.834183] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.837399] TCP: Hash tables configured (established 65536 bind 65536) [ 0.839782] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.842277] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.844627] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.847189] NET: Registered protocol family 1 [ 0.850106] RPC: Registered named UNIX socket transport module. [ 0.852809] RPC: Registered udp transport module. [ 0.854870] RPC: Registered tcp transport module. [ 0.856841] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.859403] NET: Registered protocol family 44 [ 0.861034] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.862756] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.864368] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.866142] PCI: CLS 0 bytes, default 64 [ 0.868313] Unpacking initramfs... [ 2.312370] debug: unmapping init [mem 0xffffa039fcc54000-0xffffa039fffbffff] [ 2.319366] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.321945] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.326239] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.858304] Initialise system trusted keyrings [ 2.860257] Key type blacklist registered [ 2.862270] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.875168] zbud: loaded [ 2.879644] *** VALIDATE nfs *** [ 2.881272] *** VALIDATE nfs4 *** [ 2.883353] pstore: using deflate compression [ 2.887779] Platform Keyring initialized [ 2.999972] NET: Registered protocol family 38 [ 3.001923] Key type asymmetric registered [ 3.003622] Asymmetric key parser 'x509' registered [ 3.006710] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.009782] io scheduler mq-deadline registered [ 3.011073] io scheduler kyber registered [ 3.012360] io scheduler bfq registered [ 3.014294] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.016534] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.018939] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.021297] ACPI: Power Button [PWRF] [ 3.026638] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.033735] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.050565] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.060233] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.080954] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.109207] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.137537] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.143514] Non-volatile memory driver v1.3 [ 3.145336] Linux agpgart interface v0.103 [ 3.180128] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.183459] vda: detected capacity change from 0 to 74760192 [ 3.200692] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.203649] vdb: detected capacity change from 0 to 1073741824 [ 3.219919] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.222827] vdc: detected capacity change from 0 to 2621440000 [ 3.240419] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.243607] vdd: detected capacity change from 0 to 2621440000 [ 3.269152] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.272375] vde: detected capacity change from 0 to 4294967296 [ 3.297676] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.302268] vdf: detected capacity change from 0 to 4294967296 [ 3.317637] libphy: Fixed MDIO Bus: probed [ 3.326556] usbcore: registered new interface driver usbserial_generic [ 3.329623] usbserial: USB Serial support registered for generic [ 3.332600] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.338601] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.340694] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.343561] mousedev: PS/2 mouse device common for all mice [ 3.347280] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.348322] rtc_cmos 00:05: RTC can wake from S4 [ 3.358079] rtc_cmos 00:05: registered as rtc0 [ 3.359770] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.363535] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.365613] intel_pstate: CPU model not supported [ 3.370277] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.373446] hid: raw HID events driver (C) Jiri Kosina [ 3.379681] usbcore: registered new interface driver usbhid [ 3.382252] usbhid: USB HID core driver [ 3.384273] drop_monitor: Initializing network drop monitor service [ 3.387303] Initializing XFRM netlink socket [ 3.389224] NET: Registered protocol family 10 [ 3.392394] Segment Routing with IPv6 [ 3.395016] NET: Registered protocol family 17 [ 3.396918] mpls_gso: MPLS GSO support [ 3.402233] RAS: Correctable Errors collector initialized. [ 3.404168] AVX version of gcm_enc/dec engaged. [ 3.407617] AES CTR mode by8 optimization enabled [ 3.506957] sched_clock: Marking stable (3506894966, 0)->(4412697251, -905802285) [ 3.515615] registered taskstats version 1 [ 3.518604] Loading compiled-in X.509 certificates [ 3.520936] zswap: loaded using pool lzo/zbud [ 3.551853] Key type big_key registered [ 3.569242] Key type encrypted registered [ 3.570818] ima: No TPM chip found, activating TPM-bypass! [ 3.572786] ima: Allocated hash algorithm: sha1 [ 3.574255] ima: No architecture policies found [ 3.575797] evm: Initialising EVM extended attributes: [ 3.577540] evm: security.selinux [ 3.578631] evm: security.ima [ 3.579617] evm: security.capability [ 3.580777] evm: HMAC attrs: 0x1 [ 3.584598] rtc_cmos 00:05: setting system clock to 2026-08-31 05:24:03 UTC (1788153843) [ 3.593814] debug: unmapping init [mem 0xffffffff93403000-0xffffffff935fffff] [ 3.596944] debug: unmapping init [mem 0xffffffff92182000-0xffffffff92458fff] [ 3.608257] Write protecting the kernel read-only data: 28672k [ 3.613048] debug: unmapping init [mem 0xffffffff90803000-0xffffffff909fffff] [ 3.616861] debug: unmapping init [mem 0xffffffff91114000-0xffffffff911fffff] [ 3.657768] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.668794] systemd[1]: Detected virtualization kvm. [ 3.671841] systemd[1]: Detected architecture x86-64. [ 3.674533] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.707892] systemd[1]: No hostname configured. [ 3.710644] systemd[1]: Set hostname to . [ 3.713893] random: systemd: uninitialized urandom read (16 bytes read) [ 3.717213] systemd[1]: Initializing machine ID from random generator. [ 3.782105] random: ln: uninitialized urandom read (6 bytes read) [ 3.889675] random: systemd: uninitialized urandom read (16 bytes read) [ 3.893845] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.899422] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.904637] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.564646] device-mapper: uevent: version 1.0.3 [ 4.567349] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.318690] virtio_net virtio0 ens2: renamed from eth0 [ 5.331420] random: fast init done [ 5.392700] scsi host0: ata_piix [ 5.493665] scsi host1: ata_piix [ 5.495555] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.499518] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.790483] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.087174] random: crng init done [ 10.088587] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.579959] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.845378] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.130070] SELinux: Disabled at runtime. [ 12.190250] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.203772] systemd[1]: Detected virtualization kvm. [ 12.206236] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.781574] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.785346] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.789614] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.794723] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.798153] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.805069] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.808883] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.954307] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.304327] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.578021] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.653806] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.751854] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.764358] EDAC sbridge: Ver: 1.1.2 [ 15.607641] Key type dns_resolver registered [ 15.954496] NFS: Registering the id_resolver key type [ 15.956726] Key type id_resolver registered [ 15.958589] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg657-server login: [ 65.612677] libcfs: loading out-of-tree module taints kernel. [ 65.676719] Key type ._llcrypt registered [ 65.678745] Key type .llcrypt registered [ 65.780283] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 82.818798] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 84.118174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 84.132176] alg: No test for adler32 (adler32-zlib) [ 85.729301] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 86.479157] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 88.336391] Key type lgssc registered [ 90.696154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.809732] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 120.583196] hrtimer: interrupt took 16222754 ns [ 148.318442] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 160.571164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 160.617710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.969835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 161.992895] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 162.151950] Lustre: lustre-MDT0000: new disk, initializing [ 162.255454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 162.271375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 166.742454] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.402553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.527208] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 179.571243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 179.575152] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 179.576740] Lustre: Skipped 1 previous similar message [ 179.665583] Lustre: lustre-MDT0001: new disk, initializing [ 179.781743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 179.838829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 179.855723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 184.601742] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.640958] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 200.450277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 200.915755] Lustre: lustre-OST0000: new disk, initializing [ 200.925538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 200.941240] Lustre: 8452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 201.065457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.373673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 203.402459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 203.560089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 209.784373] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.059687] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.182995] Lustre: lustre-OST0001: new disk, initializing [ 224.186300] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 224.191385] Lustre: 9525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.265249] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.127733] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.481827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 232.493919] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 232.526392] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 241.688351] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 247.023816] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 252.755481] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing check_logdir /tmp/testlogs/ [ 257.311721] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing yml_node [ 261.745556] Lustre: DEBUG MARKER: Client: 2.17.55.27 [ 264.246114] Lustre: DEBUG MARKER: MDS: 2.17.55.27 [ 266.255680] Lustre: DEBUG MARKER: OSS: 2.17.55.27 [ 267.672103] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 31 01:28:26 EDT 2026 [ 284.459775] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 285.993396] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 302.564296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 302.569212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.581785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 304.098847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 304.101380] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.109542] Lustre: Skipped 1 previous similar message [ 304.122580] Lustre: Skipped 1 previous similar message [ 307.980707] Lustre: server umount lustre-MDT0000 complete [ 311.401820] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154151 with bad export cookie 17883568383458841982 [ 311.403445] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 311.409498] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 314.337377] LustreError: 11186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 314.343652] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 314.344247] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 314.344252] Lustre: Skipped 1 previous similar message [ 314.353959] LustreError: 11186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 314.384202] Lustre: Skipped 2 previous similar messages [ 316.413552] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.445162] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 317.908854] Lustre: server umount lustre-MDT0001 complete [ 321.918090] Lustre: server umount lustre-OST0000 complete [ 325.584751] Lustre: server umount lustre-OST0001 complete [ 332.830825] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 342.978891] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 386.676810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.949223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 386.993177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 387.050691] Lustre: lustre-MDT0000: new disk, initializing [ 387.121895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 387.138416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 391.646743] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 403.074394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.238296] Lustre: 15078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 403.324142] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 403.334640] Lustre: Skipped 1 previous similar message [ 403.462065] Lustre: lustre-MDT0001: new disk, initializing [ 403.585941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 403.641433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 403.661901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 408.137980] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 418.871450] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 424.875789] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 431.630621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 431.942248] Lustre: lustre-OST0000: new disk, initializing [ 431.949486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 431.955733] Lustre: 17049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 432.041128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 433.880359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 433.892284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 434.007495] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 439.196124] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 449.032641] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 454.389539] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 454.653078] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 460.492511] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 460.706452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 464.866398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 464.875173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 464.886926] LustreError: Skipped 1 previous similar message [ 464.892528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 464.911651] Lustre: Skipped 1 previous similar message [ 464.934539] Lustre: Skipped 4 previous similar messages [ 469.156231] Lustre: server umount lustre-OST0000 complete [ 475.107118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 475.113419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.131989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 475.158349] Lustre: Skipped 1 previous similar message [ 475.183823] Lustre: Skipped 1 previous similar message [ 480.329553] Lustre: server umount lustre-MDT0000 complete [ 484.279603] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154324 with bad export cookie 17883568383458843508 [ 484.285101] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 484.304659] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 484.627769] Lustre: server umount lustre-MDT0001 complete [ 490.273781] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:32:08 (1788154328) [ 509.247765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.997457] LustreError: 19924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.131803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 514.941790] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 515.557546] LustreError: 19924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.590148] LustreError: 19924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 519.653535] LustreError: 19925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.930135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.945533] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 538.486250] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 544.691122] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 547.231287] Lustre: Failing over lustre-MDT0000 [ 547.545985] Lustre: server umount lustre-MDT0000 complete [ 550.322133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.333084] Lustre: Skipped 1 previous similar message [ 550.337243] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.337257] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 566.753441] Lustre: 3650:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788154390/real 1788154390] req@ffffa03a5915c000 x1875015291265664/t0(0) o400->MGC192.168.206.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788154406 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 566.787651] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 566.801375] LustreError: 19924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.821584] LustreError: 19924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 568.112258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.024973] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf82f32b18c78a101 [ 578.035637] Lustre: MGC192.168.206.157@tcp: Connection restored to 0@lo (at 0@lo) [ 578.384742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 578.390387] Lustre: Skipped 1 previous similar message [ 583.424263] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 583.654379] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 583.687946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 583.709847] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 593.171502] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 599.011107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 599.025787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.045293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 599.054986] Lustre: Skipped 2 previous similar messages [ 611.296147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 611.445046] Lustre: server umount lustre-MDT0000 complete [ 615.127197] LustreError: 20660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154455 with bad export cookie 17883568383458844929 [ 615.128983] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 615.145836] LustreError: 20660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 615.481263] Lustre: server umount lustre-MDT0001 complete [ 623.896528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 624.434522] LustreError: 23267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 624.444193] LustreError: 23267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 627.979633] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 635.755360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 641.142879] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 648.482811] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 653.985612] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 656.094762] Lustre: Failing over lustre-MDT0000 [ 656.435855] Lustre: server umount lustre-MDT0000 complete [ 656.869867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 656.878737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 656.893220] Lustre: Skipped 1 previous similar message [ 674.547183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.678082] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 674.929298] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 679.263425] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 679.909849] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 679.912652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 679.921288] Lustre: Skipped 1 previous similar message [ 679.936665] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 688.360747] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 695.281035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 695.281628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 695.299358] Lustre: Skipped 7 previous similar messages [ 698.362146] Lustre: server umount lustre-MDT0000 complete [ 700.389721] LustreError: 23996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 700.410571] LustreError: 23996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 701.914021] LustreError: 23247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154541 with bad export cookie 17883568383458845867 [ 701.917188] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 701.924903] LustreError: 23247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 702.277298] Lustre: server umount lustre-MDT0001 complete [ 710.704327] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:35:49 (1788154549) [ 726.767788] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 736.682690] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 779.331905] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 791.905260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 792.128883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 792.175530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 792.262754] Lustre: lustre-MDT0000: new disk, initializing [ 792.350579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 792.359023] Lustre: Skipped 3 previous similar messages [ 792.394440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 796.932412] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 808.818537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 808.913317] Lustre: 29639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 809.241181] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 814.388215] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 823.624832] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 829.703826] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 836.584755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 836.827893] Lustre: 31609:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 838.089242] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 838.146345] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 838.258528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 838.279715] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 843.167540] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 853.449669] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 859.534021] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 884.196192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 884.215698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.226026] Lustre: Skipped 3 previous similar messages [ 884.230631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 884.238265] Lustre: Skipped 1 previous similar message [ 886.595465] Lustre: server umount lustre-OST0000 complete [ 896.316658] Lustre: server umount lustre-MDT0000 complete [ 899.562972] LustreError: 32317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 899.601041] LustreError: 32317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 900.098342] LustreError: 29632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154740 with bad export cookie 17883568383458846273 [ 900.110384] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 900.117826] LustreError: 29632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 900.625966] Lustre: server umount lustre-MDT0001 complete [ 911.160725] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 914.025190] Key type lgssc unregistered [ 914.349462] LNet: 33589:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.353916] LNetError: 33589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.370357] LNet: Removed LNI 192.168.206.157@tcp [ 915.592145] Key type .llcrypt unregistered [ 915.596230] Key type ._llcrypt unregistered [ 941.643306] Key type ._llcrypt registered [ 941.645432] Key type .llcrypt registered [ 941.752245] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 942.798200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 942.816353] alg: No test for adler32 (adler32-zlib) [ 943.896245] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 944.252118] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 946.012912] Key type lgssc registered [ 947.173858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 955.213678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 955.233742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 956.693088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 961.566420] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 966.114308] LustreError: 35000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 966.131956] LustreError: 35000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 970.315646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 970.699604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 976.053415] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 983.229691] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 990.162430] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 997.106477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 997.497766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1001.605318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1003.163515] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1009.732044] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1015.927084] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1026.854642] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:41:05 (1788154865) [ 1028.738489] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1030.478753] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1036.827218] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:41:15 (1788154875) [ 1045.334648] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:41:23 (1788154883) [ 1046.410547] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1054.825135] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1054.832167] Lustre: Skipped 2 previous similar messages [ 1056.922767] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:41:34 (1788154894) [ 1060.385787] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1060.390154] Lustre: Skipped 3 previous similar messages [ 1078.048286] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1078.054915] Lustre: Skipped 6 previous similar messages [ 1080.836316] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:41:59 (1788154919) [ 1083.742842] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1087.472720] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1096.438327] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:42:14 (1788154934) [ 1098.632981] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1098.641903] Lustre: Skipped 3 previous similar messages [ 1113.085048] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:42:31 (1788154951) [ 1128.929583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1128.941412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.954212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1128.965242] Lustre: Skipped 1 previous similar message [ 1128.989531] Lustre: Skipped 1 previous similar message [ 1133.740579] Lustre: server umount lustre-OST0000 complete [ 1139.169462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.187776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1139.198062] Lustre: Skipped 1 previous similar message [ 1143.892844] Lustre: server umount lustre-MDT0000 complete [ 1144.293432] LustreError: 34996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1144.307704] LustreError: 34996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1147.776928] LustreError: 34982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154987 with bad export cookie 13541966885401055955 [ 1147.786293] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1147.791795] LustreError: 34982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1148.154418] Lustre: server umount lustre-MDT0001 complete [ 1159.629362] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 1162.762180] Key type lgssc unregistered [ 1163.088169] LNet: 42856:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.092350] LNetError: 42856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1163.112990] LNet: Removed LNI 192.168.206.157@tcp [ 1163.962288] Key type .llcrypt unregistered [ 1163.965730] Key type ._llcrypt unregistered [ 1182.986769] Key type ._llcrypt registered [ 1182.989953] Key type .llcrypt registered [ 1183.119767] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 1198.616118] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 1200.109422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1200.130128] alg: No test for adler32 (adler32-zlib) [ 1201.311437] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1201.719118] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 1203.409677] Key type lgssc registered [ 1204.708808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1241.719838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1241.773488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1243.092504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1243.120512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1243.185364] Lustre: lustre-MDT0000: new disk, initializing [ 1243.244838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1243.258336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1248.945534] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1260.608293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1260.811910] Lustre: 46007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1260.940907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1260.956155] Lustre: Skipped 1 previous similar message [ 1261.055299] Lustre: lustre-MDT0001: new disk, initializing [ 1261.181677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1261.214863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1261.235287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1266.057641] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1276.719134] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1282.439690] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1288.871085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1289.201534] Lustre: lustre-OST0000: new disk, initializing [ 1289.206640] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1289.214144] Lustre: 47975:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1289.303230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1294.976335] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1299.045786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1299.071309] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1299.146845] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1304.471493] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1309.086914] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1309.339145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1313.646575] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1313.829635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1319.913780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1319.916737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1319.922063] LustreError: Skipped 1 previous similar message [ 1319.923092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1319.941879] Lustre: Skipped 1 previous similar message [ 1321.627581] Lustre: server umount lustre-OST0000 complete [ 1327.586300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1327.599185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1327.628368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1327.637977] Lustre: Skipped 1 previous similar message [ 1330.156568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1330.162196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1331.544640] Lustre: server umount lustre-MDT0000 complete [ 1335.550583] LustreError: 46000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155175 with bad export cookie 728542768744412584 [ 1335.555109] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1335.561855] LustreError: 46000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1335.995425] Lustre: server umount lustre-MDT0001 complete [ 1346.342595] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:46:24 (1788155184) [ 1353.354597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1354.001326] LustreError: 50438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.029289] LustreError: 50438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1354.138050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.023812] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1359.339584] LustreError: 50439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.371556] LustreError: 50439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1363.430543] LustreError: 50438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1367.791458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.134949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1372.726970] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1380.386107] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1386.969611] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1394.087450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1394.582298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1401.449775] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1410.341220] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1418.473334] LustreError: 52966:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1419.570121] Lustre: Modifying parameter general.timeout=1 in log params [ 1421.459761] Lustre: Modifying parameter general.timeout=3 in log params [ 1421.464106] Lustre: Skipped 1 previous similar message [ 1423.505700] Lustre: Modifying parameter general.timeout=5 in log params [ 1423.515254] Lustre: Skipped 1 previous similar message [ 1427.630464] Lustre: Modifying parameter general.timeout=9 in log params [ 1427.635228] Lustre: Skipped 3 previous similar messages [ 1435.980892] Lustre: Modifying parameter general.timeout=17 in log params [ 1435.990658] Lustre: Skipped 7 previous similar messages [ 1452.552771] Lustre: Modifying parameter general.timeout=34 in log params [ 1452.556177] Lustre: Skipped 16 previous similar messages [ 1484.983361] Lustre: Modifying parameter general.timeout=67 in log params [ 1484.992498] Lustre: Skipped 32 previous similar messages [ 1556.416513] Lustre: Disabling parameter general.timeout= in log params [ 1556.424861] Lustre: Skipped 62 previous similar messages [ 1559.425379] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:49:57 (1788155397) [ 1563.174745] Lustre: 59918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1565.524348] Lustre: 60029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1567.785823] Lustre: 60129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1572.638310] Lustre: 60325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1572.647662] Lustre: 60325:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1604.444153] Lustre: 61800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1604.454996] Lustre: 61800:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1604.485363] LustreError: 61800:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1608.865518] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:50:47 (1788155447) [ 1623.529067] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1623.530829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1623.548492] Lustre: Skipped 1 previous similar message [ 1623.562040] Lustre: Skipped 3 previous similar messages [ 1627.200651] Lustre: server umount lustre-MDT0000 complete [ 1628.643300] LustreError: 50433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1628.665266] LustreError: 50433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1630.717584] LustreError: 51173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155470 with bad export cookie 728542768744413466 [ 1630.725770] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1630.732596] LustreError: 51173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1631.098859] Lustre: server umount lustre-MDT0001 complete [ 1645.886159] Lustre: server umount lustre-OST0000 complete [ 1675.474594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1675.762482] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1676.124672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1680.676653] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1689.814960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1689.927905] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1689.948418] Lustre: 64330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1695.726228] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1704.403575] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1710.679882] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1718.120193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1718.390971] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1718.828955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1718.834099] Lustre: Skipped 1 previous similar message [ 1725.295538] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1733.904052] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1736.245933] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1736.262960] Lustre: Skipped 23 previous similar messages [ 1736.331359] Lustre: 66898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1744.358482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1744.403078] Lustre: Skipped 2 previous similar messages [ 1744.407734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1744.419204] Lustre: Skipped 1 previous similar message [ 1750.385061] Lustre: server umount lustre-OST0000 complete [ 1756.641557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1756.664621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.700094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1756.706373] Lustre: Skipped 2 previous similar messages [ 1761.010719] Lustre: server umount lustre-MDT0000 complete [ 1764.841450] LustreError: 65283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1764.863235] LustreError: 65283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1764.922121] LustreError: 65082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155604 with bad export cookie 728542768744416651 [ 1764.938339] LustreError: 65082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1764.941293] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1765.302708] Lustre: server umount lustre-MDT0001 complete [ 1777.660606] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 1780.312557] Key type lgssc unregistered [ 1780.633581] LNet: 68108:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.642589] LNetError: 68108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1780.672776] LNet: Removed LNI 192.168.206.157@tcp [ 1781.421607] Key type .llcrypt unregistered [ 1781.423723] Key type ._llcrypt unregistered [ 1803.320377] Key type ._llcrypt registered [ 1803.323242] Key type .llcrypt registered [ 1803.413980] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 1804.417944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1804.444053] alg: No test for adler32 (adler32-zlib) [ 1805.474685] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1805.750168] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 1807.400241] Key type lgssc registered [ 1808.505583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1815.564655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1815.576889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.917102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1822.203621] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1826.273845] LustreError: 69523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1826.296989] LustreError: 69523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1831.395444] LustreError: 69524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1832.240953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1832.923322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1837.778756] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1845.435456] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1850.641917] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1856.856875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1857.393201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1863.288204] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1871.308140] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1879.358386] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1889.261939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1889.281059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1889.304964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1892.321665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1892.324220] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.358457] Lustre: Skipped 1 previous similar message [ 1894.061191] Lustre: server umount lustre-MDT0000 complete [ 1897.451151] LustreError: 70876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.462840] LustreError: 70876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1897.601485] LustreError: 69504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155737 with bad export cookie 5763024723063107772 [ 1897.610378] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1897.616388] LustreError: 69504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1898.008653] Lustre: server umount lustre-MDT0001 complete [ 1912.704250] Lustre: server umount lustre-OST0000 complete [ 1941.723441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1941.907697] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1941.932261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1942.137842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.260903] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1954.349517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1954.465343] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1954.503981] Lustre: 74398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1955.213761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1958.766994] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1965.490057] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1971.596240] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1978.482465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1978.673458] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1979.016433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1985.483638] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1993.168538] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1995.008033] Lustre: 76973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1996.932851] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2003.427357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2003.438472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.485768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2003.507115] Lustre: Skipped 2 previous similar messages [ 2008.403941] Lustre: server umount lustre-OST0000 complete [ 2014.181102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.185671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.204281] Lustre: Skipped 2 previous similar messages [ 2014.223882] Lustre: Skipped 2 previous similar messages [ 2018.504304] Lustre: server umount lustre-MDT0000 complete [ 2019.301676] LustreError: 75317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2019.331295] LustreError: 75317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2022.376302] LustreError: 74389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155862 with bad export cookie 5763024723063108619 [ 2022.380960] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2022.388921] LustreError: 74389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2022.729444] Lustre: server umount lustre-MDT0001 complete [ 2036.728320] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 2040.052321] Key type lgssc unregistered [ 2040.401553] LNet: 78183:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2040.414609] LNetError: 78183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2040.435552] LNet: Removed LNI 192.168.206.157@tcp [ 2041.557623] Key type .llcrypt unregistered [ 2041.560091] Key type ._llcrypt unregistered [ 2071.944670] Key type ._llcrypt registered [ 2071.947598] Key type .llcrypt registered [ 2072.147629] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2073.505684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2073.577728] alg: No test for adler32 (adler32-zlib) [ 2074.963392] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2075.296815] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 2077.024176] Key type lgssc registered [ 2078.547912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2087.936195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2087.975733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2089.652659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2095.620760] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2099.170626] LustreError: 79598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2099.206104] LustreError: 79598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2104.304696] LustreError: 79599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.719630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.712108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2112.113858] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2120.963560] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2127.939711] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2134.611190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2135.314781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2142.663791] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2150.449371] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2166.271932] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2167.136070] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2168.173492] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2170.221942] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2170.225647] Lustre: Skipped 1 previous similar message [ 2174.476126] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2174.482588] Lustre: Skipped 3 previous similar messages [ 2182.841704] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2182.856408] Lustre: Skipped 7 previous similar messages [ 2199.157778] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2199.172344] Lustre: Skipped 17 previous similar messages [ 2231.584664] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2231.588308] Lustre: Skipped 34 previous similar messages [ 2269.846943] Lustre: 87738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2272.947344] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 02:01:51 (1788156111) [ 2274.891803] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2277.315975] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 02:01:55 (1788156115) [ 2282.978964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2282.984362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2283.010968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2288.099218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2288.108626] Lustre: Skipped 2 previous similar messages [ 2289.021663] Lustre: server umount lustre-OST0000 complete [ 2296.801437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2296.806283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2296.812258] Lustre: Skipped 1 previous similar message [ 2296.823248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2298.338967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.483608] Lustre: server umount lustre-MDT0000 complete [ 2303.457439] LustreError: 79598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.477783] LustreError: 79598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2304.426623] LustreError: 79579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156144 with bad export cookie 7617974140773178059 [ 2304.429191] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2304.450509] LustreError: 79579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2304.848139] Lustre: server umount lustre-MDT0001 complete [ 2316.710909] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 2319.878197] Key type lgssc unregistered [ 2320.261432] LNet: 88987:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2320.278659] LNetError: 88987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2321.319568] LNet: Removed LNI 192.168.206.157@tcp [ 2322.344972] Key type .llcrypt unregistered [ 2322.349159] Key type ._llcrypt unregistered [ 2327.409614] Key type ._llcrypt registered [ 2327.413974] Key type .llcrypt registered [ 2327.525492] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2332.060773] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules [ 2333.246923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2333.368991] alg: No test for adler32 (adler32-zlib) [ 2334.459693] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2334.735403] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 2336.376148] Key type lgssc registered [ 2337.895845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2339.172199] LustreError: 90191:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2355.089392] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2361.016591] LustreError: 90191:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2364.191778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2364.231174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2365.860306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.905079] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2375.139078] LustreError: 91216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2375.147689] LustreError: 91216:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2378.634940] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 02:03:36 (1788156216) [ 2380.260505] LustreError: 91217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.642094] LustreError: 91248:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2383.654235] LustreError: 91248:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2383.660214] LustreError: 91248:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2384.025815] Lustre: server umount lustre-MDT0000 complete [ 2396.289609] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 2399.130900] Key type lgssc unregistered [ 2399.403552] LNet: 92740:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2399.411110] LNetError: 92740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2399.441199] LNet: Removed LNI 192.168.206.157@tcp [ 2400.242769] Key type .llcrypt unregistered [ 2400.253418] Key type ._llcrypt unregistered [ 2419.924688] Key type ._llcrypt registered [ 2419.928048] Key type .llcrypt registered [ 2420.074580] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2420.880667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2420.896222] alg: No test for adler32 (adler32-zlib) [ 2421.878784] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2422.130692] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 2423.832280] Key type lgssc registered [ 2425.492980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2434.596593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2434.618207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2436.243199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2440.659320] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2446.847939] LustreError: 93959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.877280] LustreError: 93959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2449.628057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2450.336333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2455.173385] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2462.913389] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2468.937995] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2476.551477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2477.317465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2484.520892] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2492.285760] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2543.193321] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 02:06:21 (1788156381) [ 2545.214452] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2547.573151] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 02:06:25 (1788156385) [ 2552.291265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2552.307901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2552.336927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2556.387309] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.388746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2556.402458] Lustre: Skipped 1 previous similar message [ 2556.411400] Lustre: Skipped 2 previous similar messages [ 2558.190168] Lustre: server umount lustre-MDT0000 complete [ 2561.516164] LustreError: 93954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.552203] LustreError: 93954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2562.712942] LustreError: 93939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156402 with bad export cookie 8578739260949113454 [ 2562.713720] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2562.743956] LustreError: 93939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2563.427771] Lustre: server umount lustre-MDT0001 complete [ 2578.021492] Lustre: server umount lustre-OST0000 complete [ 2591.584609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2592.123805] LustreError: 98249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.298638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2597.345527] LustreError: 98250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.949114] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2602.468468] LustreError: 98249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2608.884112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2609.755379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2615.129450] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2622.900567] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2628.247883] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2639.778536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2639.944473] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2639.957256] LustreError: 98237:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2639.988508] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2639.994865] LustreError: 100333:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -98 [ 2640.004845] LustreError: 100333:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [ 2640.022864] LustreError: 100333:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2640.112307] Lustre: server umount lustre-OST0000 complete [ 2640.116958] LustreError: 100333:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2648.193026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2648.334082] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2648.338937] LustreError: 98236:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2648.348083] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2648.355859] LustreError: 100795:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -98 [ 2648.363827] LustreError: 100795:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [ 2648.372050] LustreError: 100795:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2648.441693] Lustre: server umount lustre-OST0000 complete [ 2648.444836] LustreError: 100795:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2657.094781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2657.262295] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2657.271519] Lustre: Found index 0 for lustre-OST0000, updating log [ 2657.285043] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2657.350539] Lustre: lustre-OST0000: new disk, initializing [ 2657.355550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2657.363779] Lustre: 101329:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2657.586439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2663.010150] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2666.874054] Lustre: Failing over lustre-OST0000 [ 2667.048903] Lustre: server umount lustre-OST0000 complete [ 2673.132328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2673.144260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2673.145436] Lustre: Skipped 1 previous similar message [ 2673.170231] Lustre: Skipped 1 previous similar message [ 2678.248171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2678.254148] Lustre: Skipped 1 previous similar message [ 2684.386484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2684.395442] Lustre: Skipped 3 previous similar messages [ 2685.408109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2685.597060] Lustre: server umount lustre-MDT0000 complete [ 2689.508741] LustreError: 98989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2689.539708] LustreError: 98989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2689.924729] LustreError: 98231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156529 with bad export cookie 8578739260949116198 [ 2689.932211] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2689.939090] LustreError: 98231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2690.193513] Lustre: server umount lustre-MDT0001 complete [ 2701.314259] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 02:08:59 (1788156539) [ 2719.061065] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2728.959617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2729.431437] LustreError: 103674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2729.570275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2733.851984] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2742.034913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.771727] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2749.741520] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2756.309281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2757.048711] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2762.776552] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2767.278859] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2767.283656] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2767.302255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2771.081870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2771.171972] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2771.248676] Lustre: lustre-OST0001: new disk, initializing [ 2771.251948] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2771.257909] Lustre: 105994:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2771.487560] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:56 [ 2771.520114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2771.537663] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2771.543295] Lustre: Skipped 2 previous similar messages [ 2775.663656] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:01 [ 2775.679874] Lustre: lustre-OST0000: Recovery over after 0:08, of 1 clients 1 recovered and 0 were evicted. [ 2775.680107] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2775.687470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2775.694544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2776.644331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2776.653608] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2776.774640] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2777.113914] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2784.762137] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2789.345653] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2793.954484] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2793.963142] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2793.986569] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2800.027866] Lustre: server umount lustre-MDT0001 complete [ 2800.194542] LustreError: 103671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2800.216771] LustreError: 103671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2809.686749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2809.801451] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2809.805763] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2809.848439] Lustre: 103665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2811.758082] LustreError: 107578:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2811.779921] LustreError: 107578:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2811.794753] LustreError: 103667:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2811.890105] LustreError: 107583:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.157@tcp: cfg command failed: rc = -17 [ 2811.900231] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2811.900231] [ 2815.101725] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2815.984630] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2826.325852] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 02:11:04 (1788156664) [ 2844.776308] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2864.684991] Lustre: 109616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2887.779556] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2891.585878] Lustre: 110753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2979.303737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2979.317309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2979.333707] Lustre: Skipped 2 previous similar messages [ 2979.340134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2979.344747] Lustre: Skipped 4 previous similar messages [ 2984.232096] Lustre: server umount lustre-MDT0000 complete [ 2984.934911] LustreError: 103671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2984.956134] LustreError: 103671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 2988.113817] LustreError: 103656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156828 with bad export cookie 8578739260949117129 [ 2988.115954] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2988.126166] LustreError: 103656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2988.371965] Lustre: server umount lustre-MDT0001 complete [ 3001.878365] Lustre: server umount lustre-OST0000 complete [ 3005.408440] Lustre: 93270:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156829/real 1788156829] req@ffffa03a46378000 x1875017740938112/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788156845 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3005.466725] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.501774] Lustre: Skipped 3 previous similar messages [ 3005.896407] Lustre: server umount lustre-OST0001 complete [ 3011.904293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3022.814744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3032.969923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3044.273406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3062.834310] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 3073.156529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3073.171315] Lustre: lustre-MDT0000: reset Object Index mappings [ 3073.544547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3073.554671] Lustre: Skipped 1 previous similar message [ 3077.301472] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3085.858370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3085.882405] Lustre: lustre-MDT0001: reset Object Index mappings [ 3090.800936] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3094.111468] Lustre: 115848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3101.086761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3109.915277] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3111.517233] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3116.535840] LustreError: 116200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3116.580692] LustreError: 116200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3118.621367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3120.374549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3120.383932] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3120.490300] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3124.920517] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3133.022640] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3136.448754] Lustre: 117722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3194.917875] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 02:17:13 (1788157033) [ 3223.015066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3223.031477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3223.036210] Lustre: Skipped 2 previous similar messages [ 3223.052863] Lustre: Skipped 6 previous similar messages [ 3227.641694] Lustre: server umount lustre-MDT0000 complete [ 3231.669657] LustreError: 114682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157071 with bad export cookie 8578739260949164246 [ 3231.683431] LustreError: 114682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3231.687731] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3238.233211] Lustre: server umount lustre-MDT0001 complete [ 3248.321304] Lustre: server umount lustre-OST0000 complete [ 3258.871309] Lustre: server umount lustre-OST0001 complete [ 3268.174807] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 3280.615806] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 3320.180892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3320.446614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3320.483969] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3320.593892] Lustre: lustre-MDT0000: new disk, initializing [ 3320.701765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3320.707256] Lustre: Skipped 3 previous similar messages [ 3320.731089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3325.739098] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3332.835291] Lustre: server umount lustre-MDT0000 complete [ 3342.466877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.820298] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3342.837290] Lustre: 121996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3342.850970] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3347.619250] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3352.564342] Lustre: server umount lustre-MDT0000 complete [ 3362.070179] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 02:20:00 (1788157200) [ 3363.670907] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3365.469232] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 02:20:03 (1788157203) [ 3381.189058] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 3396.476677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3403.243911] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3413.363704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3413.568047] Lustre: 124048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3413.690738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3413.694404] Lustre: Skipped 1 previous similar message [ 3413.784908] Lustre: lustre-MDT0001: new disk, initializing [ 3413.941903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3413.954228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3418.570508] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3423.351787] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3429.949519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3430.342684] Lustre: lustre-OST0000: new disk, initializing [ 3430.349804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3430.378151] Lustre: 125626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3432.027525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3432.040701] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3432.128554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3437.024424] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3448.465028] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3448.624689] Lustre: lustre-OST0001: new disk, initializing [ 3448.632600] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3448.645328] Lustre: 126496:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3449.937374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3449.958392] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3450.120935] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3454.861237] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3464.721807] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3468.364158] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3880.179727] Lustre: 132927:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3880.186828] Lustre: 132927:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3880.192227] Lustre: 132927:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3880.197034] Lustre: 132927:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3880.201702] Lustre: 132927:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3880.206374] Lustre: 132927:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4182.468568] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:33:40 (1788158020) [ 4197.346461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4197.362864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4197.398674] Lustre: Skipped 3 previous similar messages [ 4197.404821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4197.409065] Lustre: Skipped 4 previous similar messages [ 4203.621826] Lustre: server umount lustre-MDT0000 complete [ 4207.125426] LustreError: 124039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158047 with bad export cookie 8578739260949234631 [ 4207.140142] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4207.149726] LustreError: 124039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4208.097682] LustreError: 127306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4208.098901] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4208.111194] LustreError: 127306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4208.143519] Lustre: Skipped 3 previous similar messages [ 4208.152200] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4208.165792] Lustre: Skipped 8 previous similar messages [ 4213.699781] Lustre: server umount lustre-MDT0001 complete [ 4223.715325] Lustre: server umount lustre-OST0000 complete [ 4233.758477] Lustre: server umount lustre-OST0001 complete [ 4240.687259] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 4252.119092] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 4296.706183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4297.212462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4297.269472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4297.383318] Lustre: lustre-MDT0000: new disk, initializing [ 4297.474711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4297.478545] Lustre: Skipped 5 previous similar messages [ 4297.502248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4303.572205] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4317.751268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4317.948719] Lustre: 135990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4318.024084] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4318.026805] Lustre: Skipped 1 previous similar message [ 4318.174847] Lustre: lustre-MDT0001: new disk, initializing [ 4318.398662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4318.420107] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4323.629323] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4334.002980] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4339.724510] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4347.358713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4347.711719] Lustre: lustre-OST0000: new disk, initializing [ 4347.718275] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4347.729534] Lustre: 137956:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4347.872940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4347.883275] Lustre: Skipped 1 previous similar message [ 4349.708293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4349.730159] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4349.952849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4354.701945] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4365.093722] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4375.798616] Lustre: lustre-MDD0000: changelog on [ 4378.875918] Lustre: lustre-MDD0001: changelog on [ 6275.212538] Lustre: lustre-MDD0001: changelog off [ 6277.344515] Lustre: 135997:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6278.301209] Lustre: lustre-MDD0000: changelog off [ 6285.281940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6285.284336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6285.296324] LustreError: Skipped 1 previous similar message [ 6285.297244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6285.297248] Lustre: Skipped 2 previous similar messages [ 6285.337415] Lustre: Skipped 2 previous similar messages [ 6288.365437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6288.371208] Lustre: Skipped 2 previous similar messages [ 6293.478525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6293.488996] Lustre: Skipped 1 previous similar message [ 6299.104510] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6299.303936] Lustre: server umount lustre-OST0000 complete [ 6303.720956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6303.734721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6303.748665] Lustre: Skipped 3 previous similar messages [ 6318.055296] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6318.362280] Lustre: server umount lustre-MDT0000 complete [ 6319.083416] LustreError: 136831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6319.110121] LustreError: 136831:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6322.487964] LustreError: 137955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160162 with bad export cookie 8578739260950059434 [ 6322.493163] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6322.509440] LustreError: 137955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6322.906139] Lustre: server umount lustre-MDT0001 complete [ 6333.546321] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 6337.070677] Key type lgssc unregistered [ 6337.454827] LNet: 147105:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6337.472889] LNetError: 147105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6337.488556] LNet: Removed LNI 192.168.206.157@tcp [ 6338.423259] Key type .llcrypt unregistered [ 6338.425124] Key type ._llcrypt unregistered [ 6361.156103] Key type ._llcrypt registered [ 6361.164542] Key type .llcrypt registered [ 6361.294495] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 6375.820850] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6377.211880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6377.226271] alg: No test for adler32 (adler32-zlib) [ 6378.283144] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6378.554495] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 6380.264193] Key type lgssc registered [ 6381.142784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6419.647643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6419.662660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6420.921140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6420.938435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6421.015340] Lustre: lustre-MDT0000: new disk, initializing [ 6421.096162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6421.110619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6425.126840] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6436.046420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6436.134288] Lustre: 150309:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6436.248270] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6436.252949] Lustre: Skipped 1 previous similar message [ 6436.274472] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6436.372818] Lustre: lustre-MDT0001: new disk, initializing [ 6436.500088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6436.549385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6436.560286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6440.606504] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6449.909764] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6455.005595] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6461.211590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6461.761778] Lustre: lustre-OST0000: new disk, initializing [ 6461.768848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6461.773712] Lustre: 152274:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6461.928750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6469.108574] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6470.271096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6470.288420] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6470.460404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6480.946740] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6488.537493] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_module obdecho/obdecho [ 6490.277467] LustreError: 153279:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6490.292925] LustreError: 153279:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6492.109515] LustreError: 153376:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6502.883621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6502.890195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6502.898701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6506.465371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6506.472052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6506.479291] Lustre: Skipped 1 previous similar message [ 6506.494950] Lustre: Skipped 1 previous similar message [ 6508.567680] Lustre: server umount lustre-MDT0000 complete [ 6511.198105] LustreError: 150301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160351 with bad export cookie 2907550963312761651 [ 6511.210850] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6511.214240] LustreError: 150301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6511.475823] Lustre: server umount lustre-MDT0001 complete [ 6523.429399] Lustre: server umount lustre-OST0000 complete [ 6543.160646] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 6552.788459] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6586.525807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6586.817968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6586.864109] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6586.996332] Lustre: lustre-MDT0000: new disk, initializing [ 6587.116356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6587.136767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6592.599134] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6603.286406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6603.388369] Lustre: 157090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6603.425828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6603.429512] Lustre: Skipped 1 previous similar message [ 6603.482803] Lustre: lustre-MDT0001: new disk, initializing [ 6603.546727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6603.573639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6603.589572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6607.630626] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6616.714517] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6622.401237] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6628.919478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6629.183441] Lustre: lustre-OST0000: new disk, initializing [ 6629.186576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6629.192465] Lustre: 159058:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6629.260557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6634.505565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6634.527352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6634.599278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6635.465556] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6645.677956] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6651.173353] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6651.573144] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6655.921940] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6656.170393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6660.069734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6660.078693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6660.086106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6660.093086] Lustre: Skipped 1 previous similar message [ 6660.103250] Lustre: Skipped 1 previous similar message [ 6665.185687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6665.198679] Lustre: Skipped 1 previous similar message [ 6672.352309] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6672.767278] Lustre: server umount lustre-OST0000 complete [ 6680.545369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6680.556449] LustreError: Skipped 1 previous similar message [ 6680.558768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6680.569597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6680.576019] Lustre: Skipped 1 previous similar message [ 6680.589213] Lustre: Skipped 3 previous similar messages [ 6682.770216] Lustre: server umount lustre-MDT0000 complete [ 6686.154821] LustreError: 159057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160526 with bad export cookie 2907550963312762610 [ 6686.159807] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6686.170193] LustreError: 159057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6686.451796] Lustre: server umount lustre-MDT0001 complete [ 6689.848842] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 03:15:28 (1788160528) [ 6691.404835] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6693.669853] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 03:15:31 (1788160531) [ 6701.516030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6702.018230] LustreError: 161472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6702.038917] LustreError: 161472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6702.117543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6706.617945] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6707.172869] LustreError: 161473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6711.267092] LustreError: 161472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6715.646113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6721.669531] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6730.039670] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6736.477373] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6743.735901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6744.140294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6744.152392] Lustre: Skipped 1 previous similar message [ 6750.017607] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6756.935886] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6767.700358] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.57@tcp (stopping) [ 6769.428892] Lustre: server umount lustre-MDT0000 complete [ 6771.170779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6771.186028] LustreError: 161473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6771.199890] Lustre: Skipped 1 previous similar message [ 6772.686477] LustreError: 161454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160612 with bad export cookie 2907550963312763485 [ 6772.688194] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6772.701115] LustreError: 161454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6776.289736] LustreError: 161467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6776.323676] LustreError: 161467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6779.138885] Lustre: server umount lustre-MDT0001 complete [ 6786.794922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6800.476354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6800.658703] LustreError: 165313:0:(llog.c:1655:llog_backup()) MGC192.168.206.157@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6800.666386] Lustre: 165313:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.157@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6802.436864] LustreError: 165318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6802.451723] LustreError: 165318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6818.875831] LustreError: 165320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6818.894164] LustreError: 165320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 6853.614812] LustreError: 165318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6853.634643] LustreError: 165318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 28 previous similar messages [ 6861.282180] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6861.287531] Lustre: 165313:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.157@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6902.260234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6905.877388] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6907.372176] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6913.292029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6918.639414] LustreError: 166009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6918.667387] LustreError: 166009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 45 previous similar messages [ 6923.229040] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6924.287571] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6931.367133] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6933.105858] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6938.250755] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6939.532980] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6960.099477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6960.100489] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6960.109516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6960.117948] Lustre: Skipped 4 previous similar messages [ 6960.126639] Lustre: Skipped 6 previous similar messages [ 6971.361835] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6971.543244] Lustre: server umount lustre-MDT0000 complete [ 6974.836122] LustreError: 163316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160814 with bad export cookie 2907550963312769946 [ 6974.841659] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6974.854941] LustreError: 163316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6975.115252] Lustre: server umount lustre-MDT0001 complete [ 6989.247595] Lustre: server umount lustre-OST0000 complete [ 7011.883272] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 7023.920297] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7057.859833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7058.142525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7058.188156] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7058.277747] Lustre: lustre-MDT0000: new disk, initializing [ 7058.343642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7058.348086] Lustre: Skipped 1 previous similar message [ 7058.361302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7061.882606] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7070.824891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7070.927548] Lustre: 170418:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7071.139605] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7074.926696] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7083.623428] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7088.722066] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7095.251079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7095.577191] Lustre: lustre-OST0000: new disk, initializing [ 7095.581521] Lustre: Skipped 1 previous similar message [ 7095.586164] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7095.592196] Lustre: Skipped 2 previous similar messages [ 7095.605229] Lustre: 172389:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7102.740735] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7103.575849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7103.586070] Lustre: Skipped 1 previous similar message [ 7103.596185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7103.731242] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7112.567812] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7118.211778] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7118.377566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7123.042799] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7123.353754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7129.569428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7129.583542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7129.595732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7129.605186] Lustre: Skipped 1 previous similar message [ 7129.610537] Lustre: Skipped 6 previous similar messages [ 7131.706128] Lustre: server umount lustre-OST0000 complete [ 7137.763330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7137.776568] LustreError: Skipped 1 previous similar message [ 7141.950522] Lustre: server umount lustre-MDT0000 complete [ 7144.938942] LustreError: 170431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7144.968956] LustreError: 170431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7145.622489] LustreError: 171214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160985 with bad export cookie 2907550963312786767 [ 7145.633866] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7145.635588] LustreError: 171214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7146.122832] Lustre: server umount lustre-MDT0001 complete [ 7155.005193] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:23:13 (1788160993) [ 7161.537082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7166.715375] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7176.010793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7181.507247] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7188.929624] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7195.620940] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7202.600744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7209.077533] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7216.280959] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7228.823538] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:24:27 (1788161067) [ 7237.100323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7237.103973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7237.108361] Lustre: Skipped 2 previous similar messages [ 7237.121761] LustreError: Skipped 1 previous similar message [ 7238.389196] Lustre: server umount lustre-OST0000 complete [ 7257.057091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7257.227660] Lustre: server umount lustre-MDT0000 complete [ 7260.930630] LustreError: 174778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161100 with bad export cookie 2907550963312787656 [ 7260.932393] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7260.958125] LustreError: 174778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7261.303171] Lustre: server umount lustre-MDT0001 complete [ 7271.227800] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7273.987509] Key type lgssc unregistered [ 7274.363297] LNet: 178528:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7274.379426] LNetError: 178528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7274.414501] LNet: Removed LNI 192.168.206.157@tcp [ 7275.445100] Key type .llcrypt unregistered [ 7275.447050] Key type ._llcrypt unregistered [ 7293.637852] Key type ._llcrypt registered [ 7293.640412] Key type .llcrypt registered [ 7293.714421] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7294.701902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7294.723881] alg: No test for adler32 (adler32-zlib) [ 7295.769898] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7296.033235] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 7297.696221] Key type lgssc registered [ 7298.689827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7306.176697] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7306.208106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7321.696374] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7326.818148] LustreError: 179771:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.157@tcp: failed processing log, type 4: rc = -110 [ 7332.000410] LustreError: 179771:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7332.012548] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7332.020650] LustreError: MGC192.168.206.157@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7332.031290] LustreError: 179771:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7332.036402] LustreError: 179771:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -22 [ 7332.045650] LustreError: 179771:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [ 7332.124147] Lustre: server umount lustre-OST0000 complete [ 7332.127538] LustreError: 179771:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7340.809166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7342.185908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7346.259599] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7352.803914] LustreError: 180269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7352.813298] LustreError: 180269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7354.895151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7355.265561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7359.707390] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7366.618843] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7371.989625] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7378.275368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7378.413762] LustreError: 182086:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7378.420969] Lustre: 182086:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.157@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7378.521706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7383.674176] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7389.928618] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7393.253920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7393.257615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7393.275222] Lustre: Skipped 2 previous similar messages [ 7393.280442] Lustre: Skipped 1 previous similar message [ 7398.369025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7398.381760] Lustre: Skipped 3 previous similar messages [ 7403.490353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7403.497680] Lustre: Skipped 2 previous similar messages [ 7406.048967] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7406.267780] Lustre: server umount lustre-MDT0000 complete [ 7408.612316] LustreError: 180998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7408.632712] LustreError: 180998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7409.873800] LustreError: 181005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161249 with bad export cookie 18296993640004827137 [ 7409.875825] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7409.887132] LustreError: 181005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7410.158455] Lustre: server umount lustre-MDT0001 complete [ 7424.227268] Lustre: server umount lustre-OST0000 complete [ 7434.494863] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 7444.343219] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7477.684152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7477.946181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7477.977358] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7478.045558] Lustre: lustre-MDT0000: new disk, initializing [ 7478.098648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7478.113464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7482.130096] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7491.648669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7491.823401] Lustre: 185611:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7491.914189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7491.926777] Lustre: Skipped 1 previous similar message [ 7492.056992] Lustre: lustre-MDT0001: new disk, initializing [ 7492.206825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7492.271149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7492.286171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7496.951559] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7507.413592] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7514.031806] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7522.550176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7522.869989] Lustre: lustre-OST0000: new disk, initializing [ 7522.873585] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7522.881415] Lustre: 187579:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7522.965937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7529.420701] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7531.072074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7531.087132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7531.238948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7539.786530] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7545.028945] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7545.334394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7551.132571] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7551.379543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7557.097828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7557.117596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7557.121688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7557.138317] Lustre: Skipped 1 previous similar message [ 7562.212417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7562.228318] Lustre: Skipped 2 previous similar messages [ 7568.352957] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7568.590510] Lustre: server umount lustre-OST0000 complete [ 7573.987943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7573.990597] LustreError: Skipped 1 previous similar message [ 7574.006651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7574.030624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7574.036590] Lustre: Skipped 2 previous similar messages [ 7577.572512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7587.296184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7587.439195] Lustre: server umount lustre-MDT0000 complete [ 7587.810425] LustreError: 185619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7587.823290] LustreError: 185619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7591.004261] LustreError: 187578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161430 with bad export cookie 18296993640004827858 [ 7591.008236] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7591.021746] LustreError: 187578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7591.285272] Lustre: server umount lustre-MDT0001 complete [ 7599.047661] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:30:37 (1788161437) [ 7614.690982] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7617.017512] Key type lgssc unregistered [ 7617.294640] LNet: 190528:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7617.299772] LNetError: 190528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7617.314267] LNet: Removed LNI 192.168.206.157@tcp [ 7618.067233] Key type .llcrypt unregistered [ 7618.069307] Key type ._llcrypt unregistered [ 7634.614390] Key type ._llcrypt registered [ 7634.623975] Key type .llcrypt registered [ 7634.769525] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7636.260099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7636.281432] alg: No test for adler32 (adler32-zlib) [ 7637.256603] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7637.581704] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 7639.288185] Key type lgssc registered [ 7640.375804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7647.388504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7647.422453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7648.935635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7653.011515] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7658.468479] LustreError: 191744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7658.487682] LustreError: 191744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7660.718033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7661.077282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7665.238551] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7673.421070] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7679.309154] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7687.812226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7688.103568] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7688.105842] LustreError: 193608:0:(llog.c:1696:llog_backup()) MGC192.168.206.157@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7688.118748] Lustre: 193608:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.157@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7688.292069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7694.659296] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7701.585349] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7704.035983] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7704.038856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7704.049154] Lustre: Skipped 1 previous similar message [ 7704.065710] Lustre: Skipped 2 previous similar messages [ 7706.083580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7706.093434] Lustre: Skipped 2 previous similar messages [ 7709.618379] Lustre: server umount lustre-MDT0000 complete [ 7711.202046] LustreError: 191744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7711.234911] LustreError: 191744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7713.024661] LustreError: 191725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161552 with bad export cookie 4918249245408982481 [ 7713.030466] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7713.031710] LustreError: 191725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7714.336372] LustreError: 192475:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000599b30cb x1875023208935168/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7714.526792] Lustre: server umount lustre-MDT0001 complete [ 7728.941403] Lustre: server umount lustre-OST0000 complete [ 7737.447651] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 7747.044827] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7783.028540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7783.307328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7783.332504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7783.392217] Lustre: lustre-MDT0000: new disk, initializing [ 7783.490954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7783.509246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7787.765969] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7799.047780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7799.313184] Lustre: 197134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7799.385633] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7799.389752] Lustre: Skipped 1 previous similar message [ 7799.394391] LustreError: 197916:0:(llog.c:1696:llog_backup()) MGC192.168.206.157@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7799.404944] LustreError: 197916:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7799.412097] Lustre: 197916:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.157@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7799.448020] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7799.467140] Lustre: Skipped 1 previous similar message [ 7799.571824] Lustre: lustre-MDT0001: new disk, initializing [ 7799.754860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7799.852611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7799.874177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7805.006249] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7817.137441] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7823.870925] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7830.362321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7830.682792] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7830.685229] Lustre: Skipped 1 previous similar message [ 7830.688462] LustreError: 199079:0:(llog.c:1696:llog_backup()) MGC192.168.206.157@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7830.695130] LustreError: 199079:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7830.699727] Lustre: 199079:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.157@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7830.703700] Lustre: 199079:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7830.719188] Lustre: lustre-OST0000: new disk, initializing [ 7830.722347] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7830.727511] Lustre: 199103:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7830.830697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7836.257714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7836.277482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7836.478530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7837.634919] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7847.712843] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7852.384865] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7852.728761] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7857.955251] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7858.230112] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7862.241607] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7862.242844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7862.260754] LustreError: Skipped 1 previous similar message [ 7862.269146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7862.271911] Lustre: Skipped 2 previous similar messages [ 7862.300725] Lustre: Skipped 1 previous similar message [ 7866.460506] Lustre: server umount lustre-OST0000 complete [ 7871.459773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7871.470156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7871.487509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7877.610119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7877.626882] LustreError: 197140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7877.628234] Lustre: Skipped 2 previous similar messages [ 7877.922308] Lustre: server umount lustre-MDT0000 complete [ 7882.587653] LustreError: 197127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161722 with bad export cookie 4918249245408983202 [ 7882.590866] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7882.607767] LustreError: 197127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7882.984109] Lustre: server umount lustre-MDT0001 complete [ 7893.356554] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:35:31 (1788161731) [ 7911.158657] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7921.415376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7921.874537] LustreError: 202233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7921.918385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7926.170776] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7927.271105] LustreError: 202234:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7934.786176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7939.660954] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7942.734735] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7950.138229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7957.532619] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7968.053824] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7968.209194] Lustre: lustre-OST0001: new disk, initializing [ 7968.213427] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7968.218806] Lustre: 204543:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7968.297274] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7968.305699] Lustre: Skipped 2 previous similar messages [ 7975.585778] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7977.591747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7977.601947] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7977.703332] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7988.398993] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7998.161929] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8000.476279] Lustre: DEBUG MARKER: ADD OST3 [ 8006.669544] loop: module loaded [ 8006.731598] loop0: detected capacity change from 0 to 204800000 [ 8006.970685] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8006.991862] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8007.031616] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8013.976577] loop0: detected capacity change from 0 to 204800000 [ 8014.657996] Lustre: DEBUG MARKER: STOP OST3 [ 8019.580366] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8019.877136] Lustre: lustre-OST0003: new disk, initializing [ 8019.890607] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8019.905156] Lustre: 206558:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8021.271853] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8021.475625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8021.492815] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8021.545178] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8023.329747] LustreError: 206666:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8024.167530] Lustre: Failing over lustre-OST0003 [ 8024.407364] Lustre: server umount lustre-OST0003 complete [ 8026.607093] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8026.607914] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8026.622081] Lustre: Skipped 1 previous similar message [ 8026.622826] LustreError: 203723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8026.622834] LustreError: 203723:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8028.519522] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8038.742170] Lustre: DEBUG MARKER: START OST3 again [ 8047.087906] LustreError: 204984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8047.107658] LustreError: 204984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 8047.565106] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8047.919872] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8047.932777] Lustre: Skipped 1 previous similar message [ 8047.946933] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8049.389565] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8049.542454] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8049.543935] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8055.373561] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8077.487234] Lustre: Failing over lustre-OST0003 [ 8077.580879] Lustre: server umount lustre-OST0003 complete [ 8078.844121] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8078.862142] Lustre: Skipped 2 previous similar messages [ 8083.946106] LustreError: 203723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8083.960542] LustreError: 203723:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8099.299917] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8099.312411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8099.321639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8099.323479] Lustre: Skipped 3 previous similar messages [ 8099.343961] Lustre: Skipped 3 previous similar messages [ 8101.589061] Lustre: server umount lustre-MDT0000 complete [ 8105.159534] LustreError: 202215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161945 with bad export cookie 4918249245408984084 [ 8105.190669] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8111.670313] Lustre: server umount lustre-MDT0001 complete [ 8115.947519] Lustre: server umount lustre-OST0000 complete [ 8120.093921] Lustre: server umount lustre-OST0001 complete [ 8127.099883] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 8136.150247] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 8177.576917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8177.832843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8177.878912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8178.021975] Lustre: lustre-MDT0000: new disk, initializing [ 8178.119861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8178.135189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8183.532515] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8194.589228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8194.725089] Lustre: 212009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8194.811930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8194.815507] Lustre: Skipped 1 previous similar message [ 8194.875864] Lustre: lustre-MDT0001: new disk, initializing [ 8194.997583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8195.013164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8199.481371] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8209.104754] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8214.801870] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8221.810715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8222.129394] Lustre: 213979:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8223.544053] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8223.803043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8230.718168] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8242.705431] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8248.718501] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8248.931703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8254.644304] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8255.003947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8259.559150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8259.566729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8259.576204] LustreError: Skipped 1 previous similar message [ 8259.593875] Lustre: Skipped 2 previous similar messages [ 8259.598760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8259.612728] Lustre: Skipped 5 previous similar messages [ 8263.483630] Lustre: server umount lustre-OST0000 complete [ 8273.554101] Lustre: server umount lustre-MDT0000 complete [ 8274.914942] LustreError: 212021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8274.929586] LustreError: 212021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 8277.231106] LustreError: 212805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162117 with bad export cookie 4918249245408986380 [ 8277.239526] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8277.249211] LustreError: 212805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8277.718562] Lustre: server umount lustre-MDT0001 complete [ 8281.739230] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:41:59 (1788162119) [ 8298.386452] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 8309.301851] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 8348.085348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8348.427300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8348.458580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8348.467384] Lustre: Skipped 1 previous similar message [ 8348.620248] Lustre: lustre-MDT0000: new disk, initializing [ 8348.624519] Lustre: Skipped 1 previous similar message [ 8348.751688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8348.759376] Lustre: Skipped 1 previous similar message [ 8353.434740] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8364.923663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8365.082784] Lustre: 218592:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8365.357650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8370.197128] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8380.158296] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8386.720298] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8394.375438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8394.691224] Lustre: 220563:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8396.723557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8400.736811] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8410.639440] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8416.848714] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8417.169489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8424.111564] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8424.489620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8427.492158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8427.508869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8427.514186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8427.528489] Lustre: Skipped 4 previous similar messages [ 8427.539100] Lustre: Skipped 3 previous similar messages [ 8432.959814] Lustre: server umount lustre-OST0000 complete [ 8444.127621] Lustre: server umount lustre-MDT0000 complete [ 8447.976199] LustreError: 218599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8448.015986] LustreError: 218599:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8448.507638] LustreError: 219396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162288 with bad export cookie 4918249245408987262 [ 8448.515406] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8448.527397] LustreError: 219396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8448.964794] Lustre: server umount lustre-MDT0001 complete [ 8461.387105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8461.965584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8461.969081] Lustre: Skipped 5 previous similar messages [ 8468.461359] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8480.225619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8485.943789] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8493.101279] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8499.746188] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8507.629945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8515.490967] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8523.793662] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8546.284121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8546.308348] LustreError: Skipped 2 previous similar messages [ 8546.325256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8546.344843] Lustre: Skipped 2 previous similar messages [ 8546.355306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8546.367294] Lustre: Skipped 5 previous similar messages [ 8550.170456] Lustre: server umount lustre-OST0000 complete [ 8559.847472] Lustre: server umount lustre-MDT0000 complete [ 8563.436502] LustreError: 222763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162403 with bad export cookie 4918249245408988144 [ 8563.445182] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8563.464583] LustreError: 222763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8563.879247] Lustre: server umount lustre-MDT0001 complete [ 8663.140660] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 8665.562535] Key type lgssc unregistered [ 8665.745362] LNet: 226470:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8665.750182] LNetError: 226470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8665.766611] LNet: Removed LNI 192.168.206.157@tcp [ 8666.848140] Key type .llcrypt unregistered [ 8666.849866] Key type ._llcrypt unregistered [ 8694.014320] Key type ._llcrypt registered [ 8694.017508] Key type .llcrypt registered [ 8694.082444] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 8709.116839] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 8710.519204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8710.562605] alg: No test for adler32 (adler32-zlib) [ 8711.685205] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8711.962824] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 8713.696174] Key type lgssc registered [ 8714.849990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8747.739057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8747.759158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8749.071349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8749.091728] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8749.145837] Lustre: lustre-MDT0000: new disk, initializing [ 8749.190746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8749.200420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8753.247049] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8763.978244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8764.189263] Lustre: 229836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8764.311106] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8764.327185] Lustre: Skipped 1 previous similar message [ 8764.448297] Lustre: lustre-MDT0001: new disk, initializing [ 8764.559361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8764.613236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8764.626801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8769.630647] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8779.425808] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8785.636893] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8792.962228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8793.304649] Lustre: lustre-OST0000: new disk, initializing [ 8793.310738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8793.322816] Lustre: 231806:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8793.423672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8798.754519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8798.767856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8798.875688] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8799.683772] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8808.129141] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8812.374760] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8812.563317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8817.319698] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8817.592193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8819.681875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8819.692962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8819.709090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8824.291964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8824.303158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8824.303828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8825.585480] Lustre: server umount lustre-OST0000 complete [ 8831.458560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8831.477538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8831.496762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8831.503181] Lustre: Skipped 1 previous similar message [ 8834.529301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8834.529759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8836.101979] Lustre: server umount lustre-MDT0000 complete [ 8840.003191] LustreError: 229830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162679 with bad export cookie 8430569336787858801 [ 8840.009325] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8840.023549] LustreError: 229830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8840.462463] Lustre: server umount lustre-MDT0001 complete [ 8863.210051] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 8875.625489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8876.163568] LustreError: 234741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8876.185214] LustreError: 234741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8876.228638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8880.321321] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8881.646842] LustreError: 234742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8886.753372] LustreError: 234741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8890.598989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8890.802620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8896.464778] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8901.684249] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8911.414122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8912.083200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8919.166327] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8928.417606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8928.706555] Lustre: lustre-OST0001: new disk, initializing [ 8928.711138] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8928.718306] Lustre: 237049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8928.862525] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8935.302458] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8936.580273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8936.600985] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8936.735643] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8946.480411] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8955.984639] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8967.753778] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:53:26 (1788162806) [ 8972.776199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8972.778125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8972.797224] Lustre: Skipped 3 previous similar messages [ 8972.814493] Lustre: Skipped 4 previous similar messages [ 8975.717388] Lustre: server umount lustre-MDT0000 complete [ 8977.895785] LustreError: 235684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8977.919787] LustreError: 235684:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8978.942254] LustreError: 234721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162818 with bad export cookie 8430569336787859683 [ 8978.950775] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8978.955450] LustreError: 234721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8979.381563] Lustre: server umount lustre-MDT0001 complete [ 8993.449772] Lustre: server umount lustre-OST0000 complete [ 9007.436450] Lustre: server umount lustre-OST0001 complete [ 9013.926590] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 9023.971307] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 9063.138693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9063.547872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9063.645906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9063.825345] Lustre: lustre-MDT0000: new disk, initializing [ 9063.965240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9063.995297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9068.750361] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9079.563228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9079.680357] Lustre: 240996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9079.756915] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9079.760460] Lustre: Skipped 1 previous similar message [ 9079.816708] Lustre: lustre-MDT0001: new disk, initializing [ 9079.937831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9079.946707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9083.660487] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9093.881754] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9100.538828] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9108.615097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9108.943461] Lustre: lustre-OST0000: new disk, initializing [ 9108.952930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9108.961957] Lustre: 242967:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9110.731253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9110.742734] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9110.886432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9116.955126] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9127.043449] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9131.901909] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9132.183903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9137.906909] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9138.261339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9141.736076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9141.748422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9141.770543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9141.775794] LustreError: Skipped 1 previous similar message [ 9141.800028] Lustre: Skipped 1 previous similar message [ 9146.487703] Lustre: server umount lustre-OST0000 complete [ 9151.458069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9156.529249] Lustre: server umount lustre-MDT0000 complete [ 9157.094097] LustreError: 241007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9157.119325] LustreError: 241007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9160.405647] LustreError: 242966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163000 with bad export cookie 8430569336787861818 [ 9160.408753] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9160.415845] LustreError: 242966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9160.945386] Lustre: server umount lustre-MDT0001 complete [ 9169.050026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9169.341025] LustreError: 245183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9169.407897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9169.417768] Lustre: Skipped 2 previous similar messages [ 9174.133682] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9184.273344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9184.740386] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9184.757147] Lustre: Skipped 2 previous similar messages [ 9190.547150] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9200.538897] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9208.666325] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9216.645373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9224.077384] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9232.148350] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9377.774982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9377.775708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9377.786809] LustreError: Skipped 1 previous similar message [ 9377.796023] Lustre: Skipped 3 previous similar messages [ 9377.798020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9377.810690] Lustre: Skipped 1 previous similar message [ 9383.939046] Lustre: server umount lustre-OST0000 complete [ 9402.336139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9402.542465] Lustre: server umount lustre-MDT0000 complete [ 9406.435163] LustreError: 245183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9406.453774] LustreError: 245183:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9406.479420] LustreError: 245164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163246 with bad export cookie 8430569336787862700 [ 9406.483922] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9406.502651] LustreError: 245164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9406.954486] Lustre: server umount lustre-MDT0001 complete [ 9502.279157] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 9505.131704] Key type lgssc unregistered [ 9505.391708] LNet: 248809:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9505.413193] LNetError: 248809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9506.470796] LNet: Removed LNI 192.168.206.157@tcp [ 9507.573261] Key type .llcrypt unregistered [ 9507.576197] Key type ._llcrypt unregistered [ 9531.596915] Key type ._llcrypt registered [ 9531.599885] Key type .llcrypt registered [ 9531.743173] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 9547.388728] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 9548.672169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9548.809505] alg: No test for adler32 (adler32-zlib) [ 9549.853653] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9550.126545] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 9551.841455] Key type lgssc registered [ 9552.972393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9595.449453] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 9604.457690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9604.472724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9605.683582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9605.719348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9605.794747] Lustre: lustre-MDT0000: new disk, initializing [ 9605.895663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9605.910604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9610.476508] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9620.758543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9620.928558] Lustre: 252877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9620.999978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9621.003383] Lustre: Skipped 1 previous similar message [ 9621.126556] Lustre: lustre-MDT0001: new disk, initializing [ 9621.275907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9621.311292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9621.334789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9626.530469] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9632.656788] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9641.094030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9641.342369] Lustre: lustre-OST0000: new disk, initializing [ 9641.347670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9641.353720] Lustre: 254507:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9641.458171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9647.875068] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9648.186797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9648.204303] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9648.364381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9658.353935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9658.553800] Lustre: lustre-OST0001: new disk, initializing [ 9658.557294] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9658.564194] Lustre: 255377:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9658.639178] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9664.686530] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9668.219822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9668.233680] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9668.285102] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9675.732477] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9682.121716] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9694.183306] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9694.192513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9694.207115] Lustre: Skipped 3 previous similar messages [ 9694.216549] Lustre: Skipped 3 previous similar messages [ 9699.299721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9699.307438] Lustre: Skipped 3 previous similar messages [ 9699.798507] Lustre: server umount lustre-MDT0000 complete [ 9702.960928] LustreError: 252871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163542 with bad export cookie 8603004140586050337 [ 9702.970299] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9702.975916] LustreError: 252871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9703.461239] Lustre: server umount lustre-MDT0001 complete [ 9716.298900] Lustre: server umount lustre-OST0000 complete [ 9729.995555] Lustre: server umount lustre-OST0001 complete [ 9743.443318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9752.257499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9761.238308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9771.190688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9790.304345] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 9800.850498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9800.884839] Lustre: lustre-MDT0000: reset Object Index mappings [ 9801.315413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9806.014419] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9806.817477] LustreError: 260393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9806.828782] LustreError: 260393:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9811.947987] LustreError: 260392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9815.323031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9815.347879] Lustre: lustre-MDT0001: reset Object Index mappings [ 9815.831188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9820.271257] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9823.474173] Lustre: 261534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9829.450181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9835.735560] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9841.066878] LustreError: 261888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9843.764295] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9843.959443] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9843.963710] Lustre: Skipped 1 previous similar message [ 9847.044816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9847.048826] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9849.883865] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9857.588745] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9861.343466] Lustre: 263408:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9867.232948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9867.242483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9867.256495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9867.418778] Lustre: server umount lustre-MDT0000 complete [ 9867.756836] LustreError: 260389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9867.767610] LustreError: 260389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9870.386332] LustreError: 263410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163710 with bad export cookie 8603004140586057120 [ 9870.391113] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9870.396672] LustreError: 263410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9870.660328] Lustre: server umount lustre-MDT0001 complete [ 9884.686512] Lustre: server umount lustre-OST0000 complete [ 9899.301970] Lustre: server umount lustre-OST0001 complete [ 9918.485552] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 04:09:16 (1788163756) [ 9933.221457] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 9942.820410] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 9979.233389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9979.489561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9979.515213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9979.607366] Lustre: lustre-MDT0000: new disk, initializing [ 9979.699386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9979.718211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9984.332687] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9994.337695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9994.480657] Lustre: 267663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9994.581087] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9994.587364] Lustre: Skipped 1 previous similar message [ 9994.665405] Lustre: lustre-MDT0001: new disk, initializing [ 9994.774657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9994.784650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9998.644196] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10008.034475] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10013.605146] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10020.138424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10020.385991] Lustre: lustre-OST0000: new disk, initializing [10020.389939] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10020.395963] Lustre: 269631:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10022.355196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10022.367118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10022.499978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10026.383978] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10035.521798] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10039.808689] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10040.034718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10044.207326] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10044.395529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10047.969448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10047.975042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10047.977867] LustreError: Skipped 1 previous similar message [10047.984961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10048.003242] Lustre: Skipped 4 previous similar messages [10048.022465] Lustre: Skipped 1 previous similar message [10053.092493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10053.098738] Lustre: Skipped 1 previous similar message [10060.256377] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10060.419488] Lustre: server umount lustre-OST0000 complete [10066.401195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10066.412054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10066.441287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10066.447397] Lustre: Skipped 2 previous similar messages [10070.289226] Lustre: server umount lustre-MDT0000 complete [10073.404472] LustreError: 267653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163913 with bad export cookie 8603004140586059220 [10073.415417] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10073.418966] LustreError: 267653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10073.666532] Lustre: server umount lustre-MDT0001 complete [10091.811818] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [10101.099549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10101.440850] LustreError: 272564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10101.483797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10101.489176] Lustre: Skipped 2 previous similar messages [10105.974342] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10111.975426] LustreError: 272565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10112.000112] LustreError: 272565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10114.195811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10119.182568] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10121.724331] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10127.760822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10133.187534] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10140.412878] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10140.613315] Lustre: lustre-OST0001: new disk, initializing [10140.618421] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10140.626292] Lustre: 274874:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10141.757478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10141.766119] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10141.940714] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10146.292943] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10155.751434] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10159.238645] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10164.450708] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10164.525384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10170.853081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10170.857105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10170.862930] Lustre: Skipped 1 previous similar message [10170.866885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10170.871763] Lustre: Skipped 2 previous similar messages [10176.569750] Lustre: server umount lustre-MDT0000 complete [10178.023936] LustreError: 272565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10178.040535] LustreError: 272565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10179.616194] LustreError: 272545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164019 with bad export cookie 8603004140586060102 [10179.625682] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10179.628521] LustreError: 272545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10179.927381] Lustre: server umount lustre-MDT0001 complete [10192.430523] Lustre: server umount lustre-OST0000 complete [10205.925161] Lustre: server umount lustre-OST0001 complete [10219.995500] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [10228.659443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10228.978050] LustreError: 277928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10232.200935] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10238.451504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10238.708811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10238.715969] Lustre: Skipped 4 previous similar messages [10242.501955] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10245.581561] Lustre: 279067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10251.153358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10253.512593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10256.520364] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10263.266462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10264.521795] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10268.654912] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10274.639869] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10277.413226] Lustre: 280944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10281.187613] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10281.281219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10288.204692] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 04:15:26 (1788164126) [10305.508407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10305.517091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10305.518347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10305.530132] Lustre: Skipped 6 previous similar messages [10305.538855] Lustre: Skipped 6 previous similar messages [10307.976536] Lustre: server umount lustre-MDT0000 complete [10310.634178] LustreError: 277928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10310.643368] LustreError: 277928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10310.720265] LustreError: 277910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164150 with bad export cookie 8603004140586066864 [10310.729758] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10317.126472] Lustre: server umount lustre-MDT0001 complete [10326.186419] Lustre: server umount lustre-OST0000 complete [10342.881581] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10343.030434] Lustre: server umount lustre-OST0001 complete [10348.044227] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [10355.870819] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [10394.761511] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [10403.030213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10403.243950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10403.274800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10403.339837] Lustre: lustre-MDT0000: new disk, initializing [10403.416894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10407.145940] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10417.047295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10417.189096] Lustre: 285181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10417.492512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10421.569936] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10431.878497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10432.167458] Lustre: 286815:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10434.247147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10437.583567] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10446.453760] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10446.621761] Lustre: 287686:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10448.245940] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10452.288685] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10460.842840] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10464.254659] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10464.260498] Lustre: Skipped 1 previous similar message [10477.508970] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 04:18:35 (1788164315) [10482.529814] Lustre: server umount lustre-MDT0000 complete [10484.204989] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10484.231153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10484.237390] LustreError: 285188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10484.237403] LustreError: 285188:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10484.243092] Lustre: Skipped 4 previous similar messages [10487.132748] LustreError: 285173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164327 with bad export cookie 8603004140586074277 [10487.143520] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10487.144017] LustreError: 285173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10487.467608] Lustre: server umount lustre-MDT0001 complete [10501.452087] Lustre: server umount lustre-OST0000 complete [10515.200581] Lustre: server umount lustre-OST0001 complete [10522.978054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10523.548438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10523.557201] Lustre: Skipped 6 previous similar messages [10527.243972] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10534.522147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10538.498448] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10545.206206] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10550.214332] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10555.511125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10561.235167] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10566.953929] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10587.015353] loop1: detected capacity change from 0 to 204800000 [10587.181164] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10587.192242] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10587.202269] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10591.425726] loop1: detected capacity change from 0 to 204800000 [10595.600339] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10595.653747] LustreError: 289922:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs21228: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10595.665083] LustreError: 289922:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs21228 [10595.673306] LustreError: 289922:0:(mgs_handler.c:517:mgs_target_reg()) fs21228-MDT0001: Error checking target index: rc = -13 [10595.683774] LustreError: fs21228-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10595.689984] LustreError: 293654:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -13 [10595.696705] LustreError: 293654:0:(tgt_mount.c:2206:server_put_super()) no obd fs21228-MDT0001 [10595.701356] LustreError: 293654:0:(tgt_mount.c:116:server_deregister_mount()) fs21228-MDT0001 not registered [10595.737884] Lustre: server umount fs21228-MDT0001 complete [10595.740852] LustreError: 293654:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10603.071982] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10603.325330] Lustre: Setting parameter fs21228-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs21228-MDT0001 [10603.406477] Lustre: srv-fs21228-MDT0001: No data found on store. Initialize space. [10603.420806] Lustre: Skipped 4 previous similar messages [10603.525199] Lustre: fs21228-MDT0001: new disk, initializing [10603.530577] Lustre: Skipped 3 previous similar messages [10603.639817] Lustre: srv-fs21228-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10603.648593] Lustre: 294120:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21228-MDT0001-mdtlov: can't create new llog: rc = -115 [10603.662609] LustreError: 294120:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21228-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10607.455932] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10613.035796] Lustre: server umount fs21228-MDT0001 complete [10624.258604] loop1: detected capacity change from 0 to 204800000 [10629.321987] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10629.617812] Lustre: 295687:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21228-MDT0001-mdtlov: can't create new llog: rc = -115 [10629.628505] LustreError: 295687:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21228-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10633.868541] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10638.428165] Lustre: server umount fs21228-MDT0001 complete [10642.924181] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10647.522577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10647.542676] Lustre: Skipped 1 previous similar message [10647.549872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10647.560862] Lustre: Skipped 2 previous similar messages [10661.344508] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10661.589944] Lustre: server umount lustre-MDT0000 complete [10669.014603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10669.127620] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10673.213854] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10674.675836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10680.165699] loop1: detected capacity change from 0 to 204800000 [10686.090150] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10686.528311] Lustre: 298137:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21228-MDT0001-mdtlov: can't create new llog: rc = -115 [10686.544752] LustreError: 298137:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21228-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10690.926207] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10696.953492] Lustre: server umount fs21228-MDT0001 complete [10718.698587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10718.711080] LustreError: Skipped 1 previous similar message [10718.713832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10718.731870] Lustre: Skipped 2 previous similar messages [10724.478895] Lustre: server umount lustre-OST0000 complete [10734.730644] Lustre: server umount lustre-MDT0000 complete [10737.604505] LustreError: 289916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164577 with bad export cookie 8603004140586076965 [10737.612326] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10737.617958] LustreError: 289916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10737.927334] Lustre: server umount lustre-MDT0001 complete [10746.284485] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [10748.557834] Key type lgssc unregistered [10748.896864] LNet: 300478:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10748.914845] LNetError: 300478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10748.934377] LNet: Removed LNI 192.168.206.157@tcp [10749.562143] Key type .llcrypt unregistered [10749.564561] Key type ._llcrypt unregistered [10766.458592] Key type ._llcrypt registered [10766.464445] Key type .llcrypt registered [10766.573673] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [10767.495432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10767.548066] alg: No test for adler32 (adler32-zlib) [10768.651481] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10768.910173] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [10770.576249] Key type lgssc registered [10771.774964] Lustre: Echo OBD driver; http://www.lustre.org/ [10778.236331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10778.255946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10779.817481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10784.422146] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10789.352776] LustreError: 301751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10789.372094] LustreError: 301751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10792.114413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10792.480306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10797.091495] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10805.398798] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10812.104574] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10819.946652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10820.238887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10824.298096] LustreError: 303580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10825.871483] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10829.804901] LustreError: 303580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10832.241358] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10833.831563] LustreError: 303582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10837.991897] LustreError: 303583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10838.007128] LustreError: 303583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10848.229077] LustreError: 304163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10848.250295] LustreError: 304163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10850.602682] loop1: detected capacity change from 0 to 204800000 [10850.725324] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10850.732250] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10850.739437] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10854.604034] loop1: detected capacity change from 0 to 204800000 [10859.454754] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10859.626740] Lustre: fs14913-OST0000: new disk, initializing [10859.633402] Lustre: srv-fs14913-OST0000: No data found on store. Initialize space. [10859.639833] Lustre: 305486:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10859.690311] Lustre: fs14913-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10864.715812] LustreError: 303582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10864.732784] LustreError: 303582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10865.585773] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10873.038951] Lustre: server umount fs14913-OST0000 complete [10884.601166] loop1: detected capacity change from 0 to 204800000 [10889.890492] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10890.099449] Lustre: fs14913-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10896.430334] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10899.427694] LustreError: 304163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10899.445269] LustreError: 304163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [10904.553098] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10905.338091] loop2: detected capacity change from 0 to 204800000 [10905.492492] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10905.516327] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10905.539224] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10910.127453] loop2: detected capacity change from 0 to 204800000 [10915.200508] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10915.296872] LustreError: fs14913-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10915.308099] LustreError: fs14913-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10915.322382] LustreError: 308527:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -13 [10915.331510] LustreError: 308527:0:(tgt_mount.c:2206:server_put_super()) no obd fs14913-OST0001 [10915.340159] LustreError: 308527:0:(tgt_mount.c:116:server_deregister_mount()) fs14913-OST0001 not registered [10915.368304] Lustre: server umount fs14913-OST0001 complete [10915.375373] LustreError: 308527:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10919.927917] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10925.293632] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10925.499922] Lustre: fs14913-OST0001: new disk, initializing [10925.504230] Lustre: srv-fs14913-OST0001: No data found on store. Initialize space. [10925.510947] Lustre: 309077:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10925.596337] Lustre: fs14913-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10931.565914] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10940.108609] Lustre: server umount fs14913-OST0000 complete [10949.794546] Lustre: server umount fs14913-OST0001 complete [10962.387476] loop1: detected capacity change from 0 to 204800000 [10965.998573] LustreError: 303582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10966.022183] LustreError: 303582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [10968.347408] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10968.614353] Lustre: fs14913-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10975.195242] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10981.277600] loop2: detected capacity change from 0 to 204800000 [10986.333752] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10992.881304] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11002.604424] Lustre: server umount fs14913-OST0000 complete [11013.413613] Lustre: server umount fs14913-OST0001 complete [11029.477639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11029.491377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11029.513384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11032.555239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11032.576555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11032.591424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11034.431655] Lustre: server umount lustre-OST0000 complete [11038.185684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11038.193573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11038.208106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11038.214517] Lustre: Skipped 1 previous similar message [11042.788894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11042.800853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11042.826148] Lustre: Skipped 1 previous similar message [11044.451727] Lustre: server umount lustre-MDT0000 complete [11048.967720] LustreError: 301730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164888 with bad export cookie 16166723826007753130 [11048.971073] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11048.995014] LustreError: 301730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11049.799867] Lustre: server umount lustre-MDT0001 complete [11062.893330] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [11067.042918] Key type lgssc unregistered [11067.589588] LNet: 314999:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11067.605474] LNetError: 314999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11067.633237] LNet: Removed LNI 192.168.206.157@tcp [11068.504609] Key type .llcrypt unregistered [11068.509550] Key type ._llcrypt unregistered [11105.793488] Key type ._llcrypt registered [11105.799752] Key type .llcrypt registered [11105.965573] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [11120.678849] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [11121.631379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11121.729619] alg: No test for adler32 (adler32-zlib) [11122.729345] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11122.952243] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [11124.626094] Key type lgssc registered [11125.698606] Lustre: Echo OBD driver; http://www.lustre.org/ [11163.453678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11163.470198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11164.763588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11164.798846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11164.928848] Lustre: lustre-MDT0000: new disk, initializing [11165.051104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11165.074312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11169.598085] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11180.416204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11180.522808] Lustre: 318861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11180.589877] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11180.596796] Lustre: Skipped 1 previous similar message [11180.763342] Lustre: lustre-MDT0001: new disk, initializing [11180.859869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11180.893325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11180.903262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11185.313397] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11195.144982] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11200.835768] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11206.849985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11207.097309] Lustre: lustre-OST0000: new disk, initializing [11207.104569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11207.116965] Lustre: 320831:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11207.271377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11213.424106] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11216.496288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11216.514423] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11216.618044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11221.954383] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11223.594912] Lustre: Mounted lustre-client [11224.539740] Lustre: Unmounted lustre-client [11233.762688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11233.772385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11233.776533] Lustre: Skipped 1 previous similar message [11233.788801] Lustre: Skipped 1 previous similar message [11238.886023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11238.903418] Lustre: Skipped 1 previous similar message [11239.364632] Lustre: server umount lustre-OST0000 complete [11242.465106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11242.473230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11242.486073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11244.003357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11248.704787] Lustre: server umount lustre-MDT0000 complete [11252.603119] LustreError: 318854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165092 with bad export cookie 17978136519988746784 [11252.609112] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11252.615222] LustreError: 318854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11252.854216] Lustre: server umount lustre-MDT0001 complete [11265.499441] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [11268.184439] Key type lgssc unregistered [11279.648671] LNet: 1 local NIs in recovery (showing 1): 192.168.206.157@tcp [11279.715084] LNet: 322989:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11279.726681] LNetError: 322989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11355.559857] LNet: Removed LNI 192.168.206.157@tcp [11356.332248] Key type .llcrypt unregistered [11356.334713] Key type ._llcrypt unregistered [11375.752816] Key type ._llcrypt registered [11375.755718] Key type .llcrypt registered [11375.869224] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [11393.830457] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [11394.829189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11395.000751] alg: No test for adler32 (adler32-zlib) [11396.206639] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11396.490290] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [11398.128594] Key type lgssc registered [11398.975384] Lustre: Echo OBD driver; http://www.lustre.org/ [11428.092133] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 04:34:26 (1788165266) [11433.367571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11433.386670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11434.587937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11434.632349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11434.743386] Lustre: lustre-MDT0000: new disk, initializing [11434.910552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11434.935123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11438.454726] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11450.940174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11451.192266] Lustre: 326211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11451.336732] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11451.344470] Lustre: Skipped 1 previous similar message [11451.477762] Lustre: lustre-MDT0001: new disk, initializing [11451.655087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11451.748396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11451.768403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11456.893421] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11467.407869] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11473.744530] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11481.421808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11481.773699] Lustre: lustre-OST0000: new disk, initializing [11481.776658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11481.804654] Lustre: 328184:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11481.927989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11488.286629] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11488.853973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11488.880228] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11488.963105] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11497.249583] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11514.855565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11514.856312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11514.863851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11514.879367] Lustre: Skipped 1 previous similar message [11514.895048] Lustre: Skipped 1 previous similar message [11517.872087] Lustre: server umount lustre-OST0000 complete [11523.557121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11523.565842] LustreError: Skipped 1 previous similar message [11523.574879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11523.591987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11525.094547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11525.099283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11527.957568] Lustre: server umount lustre-MDT0000 complete [11530.797769] LustreError: 328182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165370 with bad export cookie 9089787903106085938 [11530.799919] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11530.811111] LustreError: 328182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11531.123683] Lustre: server umount lustre-MDT0001 complete [11539.767812] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [11542.365742] Key type lgssc unregistered [11542.639758] LNet: 330091:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11542.651163] LNetError: 330091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11542.664686] LNet: Removed LNI 192.168.206.157@tcp [11543.485698] Key type .llcrypt unregistered [11543.491414] Key type ._llcrypt unregistered [11561.392299] Key type ._llcrypt registered [11561.393716] Key type .llcrypt registered [11561.502495] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [11563.151582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11563.170423] alg: No test for adler32 (adler32-zlib) [11564.137992] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11564.302794] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [11565.936159] Key type lgssc registered [11566.936195] Lustre: Echo OBD driver; http://www.lustre.org/ [11575.712537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11575.752645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11577.117590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11581.323660] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11587.558994] LustreError: 331364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11587.575435] LustreError: 331364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11589.545931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11590.071735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11594.452921] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11601.588266] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11607.185390] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11613.551188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11613.933524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11619.634475] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11626.246931] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11635.267245] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11636.592697] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11641.315607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11641.320788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11641.327158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11642.849776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11642.853932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11642.866699] Lustre: Skipped 1 previous similar message [11642.893760] Lustre: Skipped 2 previous similar messages [11644.508938] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.57@tcp (stopping) [11646.188158] Lustre: server umount lustre-MDT0000 complete [11647.970055] LustreError: 331364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11647.987177] LustreError: 331364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11649.261742] LustreError: 331345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165489 with bad export cookie 5483226182952547488 [11649.274389] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11649.283124] LustreError: 331345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11649.624825] LustreError: 331361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11649.643916] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.57@tcp (stopping) [11653.089112] LustreError: 331361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11653.095017] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11654.113723] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11654.127397] Lustre: Skipped 1 previous similar message [11655.612505] Lustre: server umount lustre-MDT0001 complete [11659.428927] Lustre: server umount lustre-OST0000 complete [11667.323743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11671.624244] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11674.153438] Lustre: 335542:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.157@tcp [11675.907640] Lustre: server umount MGS complete [11683.297590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11683.662026] LustreError: 336113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11683.694460] LustreError: 336113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11683.731026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11687.339901] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11694.054095] LustreError: 336113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11694.074789] LustreError: 336113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11695.213755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11695.514435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11699.336226] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11707.050955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11707.414457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11713.130684] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11719.954078] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11722.127839] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11727.789310] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11729.497216] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11733.985603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11734.006893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11734.038736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11734.050031] Lustre: Skipped 1 previous similar message [11736.060867] Lustre: server umount lustre-OST0000 complete [11741.156856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11741.181864] Lustre: Skipped 1 previous similar message [11747.368819] Lustre: server umount lustre-MDT0000 complete [11751.395463] LustreError: 336113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11751.410128] LustreError: 336113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11751.448440] LustreError: 336093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165591 with bad export cookie 5483226182952548328 [11751.454364] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11751.460549] LustreError: 336093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11761.796561] Lustre: server umount lustre-MDT0001 complete [11771.619662] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [11774.405579] Key type lgssc unregistered [11774.773868] LNet: 339552:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11774.783696] LNetError: 339552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11774.813169] LNet: Removed LNI 192.168.206.157@tcp [11775.641079] Key type .llcrypt unregistered [11775.644602] Key type ._llcrypt unregistered [11790.880507] Key type ._llcrypt registered [11790.882536] Key type .llcrypt registered [11790.979524] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [11803.668880] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [11804.645717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11804.663100] alg: No test for adler32 (adler32-zlib) [11805.667216] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11805.831821] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [11807.448167] Key type lgssc registered [11808.669845] Lustre: Echo OBD driver; http://www.lustre.org/ [11843.989933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11844.050634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11845.350967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11845.395065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11845.545750] Lustre: lustre-MDT0000: new disk, initializing [11845.640690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11845.675224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11849.921945] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11858.892430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11859.042427] Lustre: 342692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11859.108825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11859.113359] Lustre: Skipped 1 previous similar message [11859.190064] Lustre: lustre-MDT0001: new disk, initializing [11859.275139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11859.314548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11859.330883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11863.643863] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11872.593478] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11877.744082] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11884.818537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11885.100271] Lustre: lustre-OST0000: new disk, initializing [11885.114762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11885.137047] Lustre: 344663:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11885.306602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11891.925660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11891.944606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11892.151799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11893.636349] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11904.053212] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11908.369639] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11908.566739] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11912.182942] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11912.325068] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11917.795926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11917.810863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11917.813371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11917.821871] Lustre: Skipped 1 previous similar message [11920.376801] Lustre: server umount lustre-OST0000 complete [11925.985111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11925.990952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11926.008994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11926.013326] Lustre: Skipped 1 previous similar message [11928.036188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11928.044855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11928.061414] Lustre: Skipped 1 previous similar message [11929.897961] Lustre: server umount lustre-MDT0000 complete [11933.550712] LustreError: 342684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165773 with bad export cookie 1037965359347871407 [11933.562375] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11933.957974] Lustre: server umount lustre-MDT0001 complete [11941.361718] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 04:42:59 (1788165779) [11957.559338] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [11968.738358] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [12004.633391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12004.873535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12004.899192] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12005.005551] Lustre: lustre-MDT0000: new disk, initializing [12005.091170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12005.123508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12009.884802] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12020.125499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12020.247483] Lustre: 349420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12020.300895] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12020.305668] Lustre: Skipped 1 previous similar message [12020.402402] Lustre: lustre-MDT0001: new disk, initializing [12020.515590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12020.560360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12020.576413] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12024.678840] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12033.538427] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12038.555186] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12044.680550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12044.880705] Lustre: lustre-OST0000: new disk, initializing [12044.885671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12044.893251] Lustre: 351387:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12044.973067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12050.838965] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12051.035736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12051.065026] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12051.197145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12061.217248] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12102.632346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12102.647390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12102.657041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12102.661741] LustreError: Skipped 1 previous similar message [12102.666517] Lustre: Skipped 1 previous similar message [12107.377748] Lustre: server umount lustre-OST0000 complete [12112.871617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12112.873693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12112.903268] Lustre: Skipped 1 previous similar message [12112.911088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12112.939620] Lustre: Skipped 1 previous similar message [12117.488696] Lustre: server umount lustre-MDT0000 complete [12117.987847] LustreError: 349431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12118.016859] LustreError: 349431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12121.083287] LustreError: 349413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165960 with bad export cookie 1037965359347872289 [12121.089110] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12121.097781] LustreError: 349413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12121.403546] Lustre: server umount lustre-MDT0001 complete [12129.935045] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12132.267216] Key type lgssc unregistered [12132.527723] LNet: 353562:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12132.550042] LNetError: 353562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12132.577781] LNet: Removed LNI 192.168.206.157@tcp [12133.279133] Key type .llcrypt unregistered [12133.282298] Key type ._llcrypt unregistered [12220.062664] Key type ._llcrypt registered [12220.066323] Key type .llcrypt registered [12220.220539] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12220.627327] Key type .llcrypt unregistered [12220.634565] Key type ._llcrypt unregistered [12298.149259] Key type ._llcrypt registered [12298.150791] Key type .llcrypt registered [12298.243593] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12298.667369] Key type .llcrypt unregistered [12298.670887] Key type ._llcrypt unregistered [12324.883391] Key type ._llcrypt registered [12324.886270] Key type .llcrypt registered [12324.981521] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12325.296245] Key type .llcrypt unregistered [12325.300283] Key type ._llcrypt unregistered [12351.747515] Key type ._llcrypt registered [12351.752615] Key type .llcrypt registered [12351.887751] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12352.180351] Key type .llcrypt unregistered [12352.183911] Key type ._llcrypt unregistered [12420.656469] Key type ._llcrypt registered [12420.658569] Key type .llcrypt registered [12420.761298] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12420.983338] Key type .llcrypt unregistered [12420.986357] Key type ._llcrypt unregistered [12449.710221] Key type ._llcrypt registered [12449.721103] Key type .llcrypt registered [12449.830224] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12450.090294] Key type .llcrypt unregistered [12450.102133] Key type ._llcrypt unregistered [12478.559597] Key type ._llcrypt registered [12478.560877] Key type .llcrypt registered [12478.745923] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12479.159418] Key type .llcrypt unregistered [12479.161855] Key type ._llcrypt unregistered [12505.659783] Key type ._llcrypt registered [12505.662176] Key type .llcrypt registered [12505.752939] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12505.972379] Key type .llcrypt unregistered [12505.976134] Key type ._llcrypt unregistered [12533.041016] Key type ._llcrypt registered [12533.044436] Key type .llcrypt registered [12533.173486] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12533.444659] Key type .llcrypt unregistered [12533.448273] Key type ._llcrypt unregistered [12563.590599] Key type ._llcrypt registered [12563.591658] Key type .llcrypt registered [12563.729424] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12564.041984] Key type .llcrypt unregistered [12564.046301] Key type ._llcrypt unregistered [12622.044842] Key type ._llcrypt registered [12622.046668] Key type .llcrypt registered [12622.127479] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12622.388262] Key type .llcrypt unregistered [12622.391975] Key type ._llcrypt unregistered [12648.639145] Key type ._llcrypt registered [12648.641390] Key type .llcrypt registered [12648.825991] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12649.127282] Key type .llcrypt unregistered [12649.133661] Key type ._llcrypt unregistered [12680.812669] Key type ._llcrypt registered [12680.815361] Key type .llcrypt registered [12680.973616] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12681.352277] Key type .llcrypt unregistered [12681.354428] Key type ._llcrypt unregistered [12729.405589] Key type ._llcrypt registered [12729.407184] Key type .llcrypt registered [12729.496767] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [12744.711098] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [12745.829504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12745.853984] alg: No test for adler32 (adler32-zlib) [12746.863471] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12747.140360] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [12748.880176] Key type lgssc registered [12749.855566] Lustre: Echo OBD driver; http://www.lustre.org/ [12784.432745] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12784.454600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12785.738756] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12785.763766] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12785.868428] Lustre: tst250fs-MDT0000: new disk, initializing [12785.977635] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12785.997781] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12790.930646] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12801.204443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12801.358625] Lustre: 373775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12801.401209] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12801.409593] Lustre: Skipped 1 previous similar message [12801.519380] Lustre: tst250fs-MDT0001: new disk, initializing [12801.613948] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12801.647213] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12801.661750] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12806.442826] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12811.635668] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12819.002252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12819.289744] Lustre: tst250fs-OST0000: new disk, initializing [12819.293752] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12819.301192] Lustre: 375404:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12819.388755] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12825.287194] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12826.233329] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12826.247161] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12826.382109] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12835.112250] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12835.308284] Lustre: tst250fs-OST0001: new disk, initializing [12835.310913] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12835.315837] Lustre: 376276:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12835.435550] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12841.621947] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12843.135812] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12843.153949] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12843.279336] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12846.867401] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12849.492711] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12849.497743] Lustre: Skipped 2 previous similar messages [12868.065680] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12868.076471] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12868.089281] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12869.093798] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12869.098529] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12869.107027] Lustre: Skipped 2 previous similar messages [12869.111868] Lustre: Skipped 2 previous similar messages [12871.496539] Lustre: server umount tst250fs-MDT0000 complete [12874.209466] LustreError: 376275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166714 with bad export cookie 8399539330778049018 [12874.211127] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12874.231047] LustreError: 376275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12874.278881] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12874.303844] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (stopping) [12874.308558] Lustre: Skipped 1 previous similar message [12874.525223] Lustre: server umount tst250fs-MDT0001 complete [12883.779522] Lustre: server umount tst250fs-OST0000 complete [12887.635399] Lustre: server umount tst250fs-OST0001 complete [12907.464442] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [12915.468579] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [12946.939506] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:59:45 (1788166785) [12948.667481] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12964.704809] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [12973.960737] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [13002.408963] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12733 sec ========= 05:00:40 (1788166840) [13004.101400] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:00:42 (1788166842) === [13007.563240] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:00:45 (1788166845) === [13044.971912] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [13047.558205] Key type lgssc unregistered [13047.839489] LNet: 385425:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13047.851862] LNetError: 385425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13047.891424] LNet: Removed LNI 192.168.206.157@tcp [13048.843219] Key type .llcrypt unregistered [13048.847222] Key type ._llcrypt unregistered