[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 543051296 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002375] x2apic enabled [ 0.003000] Switched APIC routing to physical x2apic. [ 0.003021] kvm-guest: setup PV IPIs [ 0.006831] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007046] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008020] pid_max: default: 32768 minimum: 301 [ 0.011041] LSM: Security Framework initializing [ 0.012065] Yama: becoming mindful. [ 0.013041] SELinux: Initializing. [ 0.014095] *** VALIDATE selinux *** [ 0.024177] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028016] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032069] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033161] *** VALIDATE tmpfs *** [ 0.035443] *** VALIDATE proc *** [ 0.037231] *** VALIDATE cgroup *** [ 0.038016] *** VALIDATE cgroup2 *** [ 0.039298] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041115] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043041] Spectre V2 : User space: Vulnerable [ 0.044009] Speculative Store Bypass: Vulnerable [ 0.047560] debug: unmapping init [mem 0xffffffffb0459000-0xffffffffb0460fff] [ 0.049224] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050776] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051034] ... version: 2 [ 0.052019] ... bit width: 48 [ 0.053013] ... generic registers: 4 [ 0.054015] ... value mask: 0000ffffffffffff [ 0.055020] ... max period: 00007fffffffffff [ 0.056018] ... fixed-purpose events: 3 [ 0.057017] ... event mask: 000000070000000f [ 0.058392] rcu: Hierarchical SRCU implementation. [ 0.060725] smp: Bringing up secondary CPUs ... [ 0.061619] x86: Booting SMP configuration: [ 0.062056] .... node #0, CPUs: #1 #2 #3 [ 0.065317] smp: Brought up 1 node, 4 CPUs [ 0.067012] smpboot: Max logical packages: 1 [ 0.068019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.154036] node 0 deferred pages initialised in 82ms [ 0.163012] devtmpfs: initialized [ 0.164303] x86/mm: Memory block size: 128MB [ 0.167885] gcov: version magic: 0x41383552 [ 0.171170] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.174155] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.177476] pinctrl core: initialized pinctrl subsystem [ 0.179182] [ 0.179779] ************************************************************* [ 0.182018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185015] ** ** [ 0.187015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.189019] ** ** [ 0.191014] ** This means that this kernel is built to expose internal ** [ 0.194014] ** IOMMU data structures, which may compromise security on ** [ 0.196013] ** your system. ** [ 0.199019] ** ** [ 0.201015] ** If you see this message and you are not debugging the ** [ 0.203017] ** kernel, report this immediately to your vendor! ** [ 0.205013] ** ** [ 0.207016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.209012] ************************************************************* [ 0.212875] NET: Registered protocol family 16 [ 0.216525] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.219089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.222071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.225123] cpuidle: using governor menu [ 0.231413] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.234460] PCI: Using configuration type 1 for base access [ 0.236138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.247227] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.248000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.253531] cryptd: max_cpu_qlen set to 1000 [ 0.256304] ACPI: Added _OSI(Module Device) [ 0.258016] ACPI: Added _OSI(Processor Device) [ 0.259015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.261013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.265528] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.273626] ACPI: Interpreter enabled [ 0.274016] ACPI: PM: (supports S0 S3 S4 S5) [ 0.275013] ACPI: Using IOAPIC for interrupt routing [ 0.276154] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.279523] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.290075] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.292043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.294066] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.298101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.303387] acpiphp: Slot [2] registered [ 0.305141] acpiphp: Slot [5] registered [ 0.306181] acpiphp: Slot [6] registered [ 0.308104] acpiphp: Slot [7] registered [ 0.309119] acpiphp: Slot [8] registered [ 0.311098] acpiphp: Slot [9] registered [ 0.312000] acpiphp: Slot [10] registered [ 0.312000] acpiphp: Slot [3] registered [ 0.314090] acpiphp: Slot [4] registered [ 0.316127] acpiphp: Slot [11] registered [ 0.318134] acpiphp: Slot [12] registered [ 0.319096] acpiphp: Slot [13] registered [ 0.320103] acpiphp: Slot [14] registered [ 0.322105] acpiphp: Slot [15] registered [ 0.323140] acpiphp: Slot [16] registered [ 0.325122] acpiphp: Slot [17] registered [ 0.326105] acpiphp: Slot [18] registered [ 0.327127] acpiphp: Slot [19] registered [ 0.329107] acpiphp: Slot [20] registered [ 0.330109] acpiphp: Slot [21] registered [ 0.332107] acpiphp: Slot [22] registered [ 0.333141] acpiphp: Slot [23] registered [ 0.335105] acpiphp: Slot [24] registered [ 0.336099] acpiphp: Slot [25] registered [ 0.337098] acpiphp: Slot [26] registered [ 0.338128] acpiphp: Slot [27] registered [ 0.340092] acpiphp: Slot [28] registered [ 0.341091] acpiphp: Slot [29] registered [ 0.342120] acpiphp: Slot [30] registered [ 0.343124] acpiphp: Slot [31] registered [ 0.345091] PCI host bridge to bus 0000:00 [ 0.346020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.348026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.350031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.353030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.355028] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.357042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.358000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.359000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.362239] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.369023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.374036] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.377025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.379024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.382026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.384638] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.387632] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.389040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.391704] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.406020] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.410019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.415559] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.428026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.445020] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.455347] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.462023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.467017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.483029] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.495620] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.501019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.508021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.521018] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.530000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.537024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.543018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.560028] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.570277] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.579022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.585024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.602020] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.612000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.614020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.618023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.634024] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.647139] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.649425] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.651389] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.653386] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.656263] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.663144] iommu: Default domain type: Passthrough [ 0.664495] SCSI subsystem initialized [ 0.666130] ACPI: bus type USB registered [ 0.668128] usbcore: registered new interface driver usbfs [ 0.670088] usbcore: registered new interface driver hub [ 0.671087] usbcore: registered new device driver usb [ 0.673202] pps_core: LinuxPPS API ver. 1 registered [ 0.674020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.677092] PTP clock support registered [ 0.683149] EDAC MC: Ver: 3.0.0 [ 0.685120] PCI: Using ACPI for IRQ routing [ 0.688000] NetLabel: Initializing [ 0.689011] NetLabel: domain hash size = 128 [ 0.691011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.693134] NetLabel: unlabeled traffic allowed by default [ 0.696174] vgaarb: loaded [ 0.699362] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.701014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.711396] clocksource: Switched to clocksource kvm-clock [ 0.823927] VFS: Disk quotas dquot_6.6.0 [ 0.825606] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.828301] *** VALIDATE ramfs *** [ 0.829350] *** VALIDATE hugetlbfs *** [ 0.830945] pnp: PnP ACPI init [ 0.833285] pnp: PnP ACPI: found 6 devices [ 0.850667] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.853838] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.855961] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.858476] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.860773] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.862599] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.865370] NET: Registered protocol family 2 [ 0.868191] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.873571] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.877886] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.883336] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.887180] TCP: Hash tables configured (established 65536 bind 65536) [ 0.889854] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.892875] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.895879] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.898943] NET: Registered protocol family 1 [ 0.901459] RPC: Registered named UNIX socket transport module. [ 0.903535] RPC: Registered udp transport module. [ 0.905435] RPC: Registered tcp transport module. [ 0.907279] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.909761] NET: Registered protocol family 44 [ 0.911509] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.913834] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.917241] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.919786] PCI: CLS 0 bytes, default 64 [ 0.921233] Unpacking initramfs... [ 2.386558] debug: unmapping init [mem 0xffff9c0dfcc54000-0xffff9c0dfffbffff] [ 2.390673] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.392951] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.395476] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.914662] Initialise system trusted keyrings [ 2.916336] Key type blacklist registered [ 2.918158] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.927712] zbud: loaded [ 2.931635] *** VALIDATE nfs *** [ 2.933047] *** VALIDATE nfs4 *** [ 2.939091] pstore: using deflate compression [ 2.943174] Platform Keyring initialized [ 3.098627] NET: Registered protocol family 38 [ 3.100355] Key type asymmetric registered [ 3.102348] Asymmetric key parser 'x509' registered [ 3.107256] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.110692] io scheduler mq-deadline registered [ 3.114052] io scheduler kyber registered [ 3.115969] io scheduler bfq registered [ 3.118563] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.122363] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.125320] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.128289] ACPI: Power Button [PWRF] [ 3.233862] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.334881] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.539568] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.652754] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.880382] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.910099] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.941917] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.947436] Non-volatile memory driver v1.3 [ 3.949277] Linux agpgart interface v0.103 [ 3.986109] virtio_blk virtio1: [vda] 133776 512-byte logical blocks (68.5 MB/65.3 MiB) [ 3.988873] vda: detected capacity change from 0 to 68493312 [ 4.008582] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.011549] vdb: detected capacity change from 0 to 1073741824 [ 4.033217] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.036870] vdc: detected capacity change from 0 to 2621440000 [ 4.054575] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.057686] vdd: detected capacity change from 0 to 2621440000 [ 4.076331] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.079358] vde: detected capacity change from 0 to 4294967296 [ 4.099954] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.103017] vdf: detected capacity change from 0 to 4294967296 [ 4.109511] libphy: Fixed MDIO Bus: probed [ 4.117402] usbcore: registered new interface driver usbserial_generic [ 4.119872] usbserial: USB Serial support registered for generic [ 4.122876] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.129430] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.131355] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.135591] mousedev: PS/2 mouse device common for all mice [ 4.138558] rtc_cmos 00:05: RTC can wake from S4 [ 4.141802] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.148271] rtc_cmos 00:05: registered as rtc0 [ 4.152786] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.156849] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.157606] intel_pstate: CPU model not supported [ 4.166277] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.168026] hid: raw HID events driver (C) Jiri Kosina [ 4.173896] usbcore: registered new interface driver usbhid [ 4.176341] usbhid: USB HID core driver [ 4.178115] drop_monitor: Initializing network drop monitor service [ 4.181195] Initializing XFRM netlink socket [ 4.183635] NET: Registered protocol family 10 [ 4.187093] Segment Routing with IPv6 [ 4.188423] NET: Registered protocol family 17 [ 4.192092] mpls_gso: MPLS GSO support [ 4.198466] RAS: Correctable Errors collector initialized. [ 4.201046] AVX version of gcm_enc/dec engaged. [ 4.203351] AES CTR mode by8 optimization enabled [ 4.315754] sched_clock: Marking stable (4315671820, 0)->(5234415725, -918743905) [ 4.319548] registered taskstats version 1 [ 4.322742] Loading compiled-in X.509 certificates [ 4.325115] zswap: loaded using pool lzo/zbud [ 4.360295] Key type big_key registered [ 4.373978] Key type encrypted registered [ 4.375815] ima: No TPM chip found, activating TPM-bypass! [ 4.379074] ima: Allocated hash algorithm: sha1 [ 4.380843] ima: No architecture policies found [ 4.384567] evm: Initialising EVM extended attributes: [ 4.386538] evm: security.selinux [ 4.391269] evm: security.ima [ 4.392536] evm: security.capability [ 4.393923] evm: HMAC attrs: 0x1 [ 4.396445] rtc_cmos 00:05: setting system clock to 2025-10-23 16:36:19 UTC (1761237379) [ 4.407089] debug: unmapping init [mem 0xffffffffb1403000-0xffffffffb15fffff] [ 4.413022] debug: unmapping init [mem 0xffffffffb0182000-0xffffffffb0458fff] [ 4.421760] Write protecting the kernel read-only data: 28672k [ 4.425925] debug: unmapping init [mem 0xffffffffae803000-0xffffffffae9fffff] [ 4.429181] debug: unmapping init [mem 0xffffffffaf114000-0xffffffffaf1fffff] [ 4.464855] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.473211] systemd[1]: Detected virtualization kvm. [ 4.474505] systemd[1]: Detected architecture x86-64. [ 4.475903] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.503316] systemd[1]: No hostname configured. [ 4.504635] systemd[1]: Set hostname to . [ 4.506455] random: systemd: uninitialized urandom read (16 bytes read) [ 4.508734] systemd[1]: Initializing machine ID from random generator. [ 4.535977] random: ln: uninitialized urandom read (6 bytes read) [ 4.655646] random: systemd: uninitialized urandom read (16 bytes read) [ 4.661347] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.667095] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.675825] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.435520] device-mapper: uevent: version 1.0.3 [ 5.437634] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.187647] random: fast init done [ 6.208028] virtio_net virtio0 ens2: renamed from eth0 [ 6.255265] scsi host0: ata_piix [ 6.275602] scsi host1: ata_piix [ 6.277162] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.279625] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.900233] random: crng init done [ 10.902151] random: 7 urandom warning(s) missed due to ratelimiting [ 10.904388] dracut-initqueue[595]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.753164] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.441327] printk: systemd: 25 output lines suppressed due to ratelimiting [ 13.827274] SELinux: Disabled at runtime. [ 13.903289] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.915235] systemd[1]: Detected virtualization kvm. [ 13.917478] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.689581] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.695344] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.704255] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.710321] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.713656] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.730124] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.736713] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ 15.132555] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 15.732897] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.489509] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 16.494438] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.775333] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 16.820250] EDAC sbridge: Ver: 1.1.2 [ 19.329577] Key type dns_resolver registered [ 19.710690] NFS: Registering the id_resolver key type [ 19.712511] Key type id_resolver registered [ 19.715785] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg613-server login: [ 103.922680] libcfs: loading out-of-tree module taints kernel. [ 103.940939] Key type ._llcrypt registered [ 103.942588] Key type .llcrypt registered [ 103.992898] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 115.928501] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 117.074938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 117.093081] alg: No test for adler32 (adler32-zlib) [ 118.284178] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 118.778494] LNet: Added LNI 192.168.206.113@tcp [8/256/0/180] [ 120.487198] Key type lgssc registered [ 121.956571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 133.739047] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 134.955935] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 142.112466] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 149.162866] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 156.771116] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 161.344298] hrtimer: interrupt took 4433356 ns [ 178.162039] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 192.444276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 192.524386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 192.540680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.819087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.866364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 193.976203] Lustre: lustre-MDT0000: new disk, initializing [ 194.156304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.216675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 200.306493] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.977940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 221.102804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 221.228298] Lustre: 6514:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 221.314973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 221.317478] Lustre: Skipped 1 previous similar message [ 221.650407] Lustre: lustre-MDT0001: new disk, initializing [ 221.842909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 221.927705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 221.953332] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 228.696697] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.710426] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 246.679310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 246.766232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 247.129202] Lustre: lustre-OST0000: new disk, initializing [ 247.135191] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 247.201744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 254.349185] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.072041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 256.095387] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 256.211319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 270.591316] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 270.665685] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 270.735414] Lustre: lustre-OST0001: new disk, initializing [ 270.738597] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 270.825103] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 276.593197] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 281.120874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 281.132632] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 281.202095] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 290.975270] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 299.979106] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 310.776573] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing check_logdir /tmp/testlogs/ [ 314.816704] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing yml_node [ 319.054199] Lustre: DEBUG MARKER: Client: 2.16.59.36 [ 321.532180] Lustre: DEBUG MARKER: MDS: 2.16.59.36 [ 324.021489] Lustre: DEBUG MARKER: OSS: 2.16.59.36 [ 325.456650] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Oct 23 12:41:39 EDT 2025 [ 341.681871] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 343.200286] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 357.859560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.863802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 357.882142] Lustre: Skipped 1 previous similar message [ 357.900826] Lustre: Skipped 3 previous similar messages [ 362.980586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 362.989575] Lustre: Skipped 3 previous similar messages [ 363.887672] LustreError: 12229:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 363.999946] Lustre: server umount lustre-MDT0000 complete [ 367.828497] LustreError: 6506:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761237742 with bad export cookie 11149990281912713613 [ 367.829069] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 367.839589] LustreError: 6506:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 368.096193] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.107261] Lustre: Skipped 2 previous similar messages [ 368.114762] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 368.122935] Lustre: Skipped 1 previous similar message [ 372.131079] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 372.133920] Lustre: Skipped 1 previous similar message [ 374.055676] LustreError: 12432:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 374.064340] LustreError: 12432:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 374.264627] Lustre: server umount lustre-MDT0001 complete [ 378.061403] LustreError: 12635:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 378.074159] LustreError: 12635:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 378.129607] Lustre: server umount lustre-OST0000 complete [ 381.597325] LustreError: 12835:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 381.606711] LustreError: 12835:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 381.914523] Lustre: server umount lustre-OST0001 complete [ 387.445988] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 395.860077] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 404.465407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 409.906668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 415.260057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 421.127567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 427.436539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 427.491938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 427.758295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 427.785459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 427.839591] Lustre: lustre-MDT0000: new disk, initializing [ 427.925937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 427.936145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 431.773831] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 441.187210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 441.268745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 441.358572] Lustre: 15020:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 441.416913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 441.419893] Lustre: Skipped 1 previous similar message [ 441.483506] Lustre: lustre-MDT0001: new disk, initializing [ 441.599548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 441.638779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 441.646346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 441.706934] Lustre: 15818:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 441.718298] Lustre: 15818:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 441.732177] Lustre: 15818:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 441.746077] Lustre: 15818:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 441.757251] Lustre: 15818:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 441.761756] Lustre: 15818:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 444.908026] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 451.879517] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 455.868271] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 462.332605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 462.398860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 462.602663] Lustre: lustre-OST0000: new disk, initializing [ 462.609457] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 462.695852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 464.170896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 464.191371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 464.307962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 468.248351] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 474.864197] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 478.175305] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 478.328803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 481.846508] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 482.016553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 484.833945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 484.843200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 484.864034] Lustre: Skipped 1 previous similar message [ 484.866785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 484.871050] Lustre: Skipped 1 previous similar message [ 495.085151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 495.088093] Lustre: Skipped 2 previous similar messages [ 498.143595] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 498.181734] LustreError: 17961:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 498.192368] LustreError: 17961:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 498.322208] Lustre: server umount lustre-OST0000 complete [ 503.267709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 503.277869] LustreError: Skipped 1 previous similar message [ 503.284357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.290323] Lustre: Skipped 1 previous similar message [ 507.779775] LustreError: 18164:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 507.793431] LustreError: 18164:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 507.915054] Lustre: server umount lustre-MDT0000 complete [ 510.437754] LustreError: 16174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.447538] LustreError: 16174:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 511.362554] LustreError: 15011:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761237886 with bad export cookie 11149990281912715188 [ 511.372562] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 511.373758] LustreError: 15011:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 511.741807] Lustre: server umount lustre-MDT0001 complete [ 516.457795] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 12:44:50 (1761237890) [ 531.755218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 532.157314] LustreError: 19500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.226890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.542634] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 537.574484] LustreError: 19500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.689200] LustreError: 19500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.094860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 546.918978] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 552.076711] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 555.760602] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 557.654493] Lustre: Failing over lustre-MDT0000 [ 557.798784] LustreError: 20844:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 557.804245] LustreError: 20844:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 557.916851] Lustre: server umount lustre-MDT0000 complete [ 559.076444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 559.083552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.090305] Lustre: Skipped 1 previous similar message [ 559.118243] LustreError: 20213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.270836] LustreError: 19495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.287704] LustreError: 19495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 575.883103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.955931] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 579.264796] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 581.602747] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 581.606059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 581.641507] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 585.780646] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 591.840473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 591.847430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 591.878171] Lustre: Skipped 1 previous similar message [ 591.887927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 591.890014] Lustre: Skipped 4 previous similar messages [ 595.699473] LustreError: 21823:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 595.703138] LustreError: 21823:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 595.801263] Lustre: server umount lustre-MDT0000 complete [ 599.015759] LustreError: 19500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.038955] LustreError: 19500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 599.435900] LustreError: 20845:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761237974 with bad export cookie 11149990281912716644 [ 599.444959] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 599.452753] LustreError: 20845:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 599.789546] Lustre: server umount lustre-MDT0001 complete [ 607.653176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.198144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 608.202118] Lustre: Skipped 2 previous similar messages [ 611.855966] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 620.608083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 624.923522] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 630.791451] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 634.655302] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 636.376248] Lustre: Failing over lustre-MDT0000 [ 636.386717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 636.395929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 636.413951] Lustre: Skipped 1 previous similar message [ 636.417428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 636.422952] Lustre: Skipped 2 previous similar messages [ 636.560458] Lustre: server umount lustre-MDT0000 complete [ 638.435077] LustreError: 22503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.449836] LustreError: 22503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 654.092097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 654.179187] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 654.531569] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 657.873467] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 659.434669] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 659.438376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 659.455888] Lustre: Skipped 1 previous similar message [ 659.482049] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 664.541779] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 669.664318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 669.670014] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.682035] Lustre: Skipped 1 previous similar message [ 674.404203] LustreError: 24831:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 674.408611] LustreError: 24831:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 674.472204] Lustre: server umount lustre-MDT0000 complete [ 677.407744] LustreError: 22482:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238052 with bad export cookie 11149990281912717575 [ 677.418671] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 677.744347] Lustre: server umount lustre-MDT0001 complete [ 684.721813] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 12:47:39 (1761238059) [ 697.821111] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 704.877503] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 713.693658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 719.183578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 724.002169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 729.260614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 737.513985] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 748.271654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 748.389912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 748.571126] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 748.594474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 748.653547] Lustre: lustre-MDT0000: new disk, initializing [ 748.742324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 748.746716] Lustre: Skipped 2 previous similar messages [ 748.762516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 752.240560] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 762.043587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 762.139627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 762.251541] Lustre: 28504:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 762.525536] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 762.658083] Lustre: 29300:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 762.679895] Lustre: 29300:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 762.687182] Lustre: 29300:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 762.701942] Lustre: 29300:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.707219] Lustre: 29300:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 762.726625] Lustre: 29300:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.730893] Lustre: 29300:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 762.740652] Lustre: 29300:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.749107] Lustre: 29300:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 762.753120] Lustre: 29300:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.756900] Lustre: 29300:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 762.762842] Lustre: 29300:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.251679] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 773.093812] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 776.035940] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 782.111552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 782.166095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 783.810289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 783.862244] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 783.926656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 783.941758] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 787.198202] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 794.010691] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 794.909354] Lustre: 28511:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 511, rollback = 2 [ 794.924072] Lustre: 28511:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 794.934154] Lustre: 28511:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/6, destroy: 0/0/0 [ 794.938477] Lustre: 28511:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.943265] Lustre: 28511:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 5/511/0 [ 794.948956] Lustre: 28511:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.957306] Lustre: 28511:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 5/58/0, punch: 0/0/0, quota 1/3/0 [ 794.965428] Lustre: 28511:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.969991] Lustre: 28511:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 5/102/3, delete: 0/0/0 [ 794.973814] Lustre: 28511:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.987990] Lustre: 28511:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 794.995492] Lustre: 28511:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 802.085897] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 819.682872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 819.686149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.691239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 819.709113] Lustre: Skipped 2 previous similar messages [ 819.717563] Lustre: Skipped 4 previous similar messages [ 824.913114] LustreError: 31164:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 824.915397] LustreError: 31164:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 825.068833] Lustre: server umount lustre-OST0000 complete [ 834.397516] Lustre: server umount lustre-MDT0000 complete [ 835.042709] LustreError: 28514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 835.054854] LustreError: 28514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 837.478042] LustreError: 28496:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238212 with bad export cookie 11149990281912717974 [ 837.485737] LustreError: 28496:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 837.491130] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 837.908907] Lustre: server umount lustre-MDT0001 complete [ 849.125374] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 851.760570] Key type lgssc unregistered [ 852.036745] LNet: 32101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.047518] LNetError: 32101:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.068087] LNet: Removed LNI 192.168.206.113@tcp [ 852.876150] Key type .llcrypt unregistered [ 852.878568] Key type ._llcrypt unregistered [ 875.139670] Key type ._llcrypt registered [ 875.141819] Key type .llcrypt registered [ 875.223766] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 876.068930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 876.138177] alg: No test for adler32 (adler32-zlib) [ 877.212540] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 877.374396] LNet: Added LNI 192.168.206.113@tcp [8/256/0/180] [ 879.015254] Key type lgssc registered [ 880.096924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 886.953081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 886.978657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.418227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 892.080431] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 898.531150] LustreError: 33492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 898.558213] LustreError: 33492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 899.513757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 899.806163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 903.231652] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 908.142788] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 911.086638] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 917.665410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 918.014177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 923.344315] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 924.164201] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 928.857773] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 945.019804] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 12:51:58 (1761238318) [ 946.768381] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 948.405981] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 955.069276] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 12:52:09 (1761238329) [ 963.397856] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 12:52:17 (1761238337) [ 964.238611] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 972.049918] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 972.054841] Lustre: Skipped 2 previous similar messages [ 973.551179] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 12:52:27 (1761238347) [ 976.984895] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 976.989353] Lustre: Skipped 3 previous similar messages [ 995.015195] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 995.027913] Lustre: Skipped 6 previous similar messages [ 997.496278] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 12:52:51 (1761238371) [ 999.884489] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1003.156600] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1010.458905] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 12:53:04 (1761238384) [ 1011.946726] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1011.961680] Lustre: Skipped 3 previous similar messages [ 1024.032518] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 12:53:18 (1761238398) [ 1035.747182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1035.750764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.761819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1040.763131] LustreError: 40064:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1040.930988] Lustre: server umount lustre-OST0000 complete [ 1045.990410] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1045.996256] Lustre: Skipped 1 previous similar message [ 1046.006038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1046.012249] Lustre: Skipped 2 previous similar messages [ 1051.106338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1051.112175] Lustre: Skipped 1 previous similar message [ 1055.204032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1055.208716] Lustre: Skipped 1 previous similar message [ 1058.271141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1058.345335] LustreError: 40266:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1058.351279] LustreError: 40266:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1058.422816] Lustre: server umount lustre-MDT0000 complete [ 1060.322869] LustreError: 35477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1060.340840] LustreError: 35477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1061.713903] LustreError: 33472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238436 with bad export cookie 6705824354286044200 [ 1061.719153] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1061.731759] LustreError: 33472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1061.839336] LustreError: 40468:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1061.842318] LustreError: 40468:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1062.071186] Lustre: server umount lustre-MDT0001 complete [ 1071.580639] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 1074.041671] Key type lgssc unregistered [ 1074.327091] LNet: 41000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1074.332644] LNetError: 41000:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1074.357994] LNet: Removed LNI 192.168.206.113@tcp [ 1075.164954] Key type .llcrypt unregistered [ 1075.168963] Key type ._llcrypt unregistered [ 1089.803687] Key type ._llcrypt registered [ 1089.805391] Key type .llcrypt registered [ 1089.916508] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 1104.181484] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 1105.005421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1105.136428] alg: No test for adler32 (adler32-zlib) [ 1106.143293] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 1106.335843] LNet: Added LNI 192.168.206.113@tcp [8/256/0/180] [ 1107.967292] Key type lgssc registered [ 1108.916177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1116.766981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1122.526980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1128.655887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1134.739835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1140.429163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1140.479433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1140.490339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1141.722939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1141.762119] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1141.860558] Lustre: lustre-MDT0000: new disk, initializing [ 1142.035912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1142.062272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1145.849298] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1155.813757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1155.865192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1155.941740] Lustre: 44126:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1156.001743] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1156.005147] Lustre: Skipped 1 previous similar message [ 1156.084319] Lustre: lustre-MDT0001: new disk, initializing [ 1156.186378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1156.230655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1156.246085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1156.331042] Lustre: 44923:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 1156.336251] Lustre: 44923:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 1156.342934] Lustre: 44923:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 1156.347434] Lustre: 44923:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 1156.361507] Lustre: 44923:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 1156.368560] Lustre: 44923:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 1159.700105] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1166.137228] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1169.487978] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1176.500940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1176.564742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1176.800227] Lustre: lustre-OST0000: new disk, initializing [ 1176.804620] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1176.903378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.880756] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1184.834904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1184.852961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1184.940405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1188.451460] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1191.826858] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1192.079062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1195.232392] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1195.409609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1200.614422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1200.622327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.638492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1200.641885] Lustre: Skipped 1 previous similar message [ 1205.735262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1210.851929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1210.858280] Lustre: Skipped 2 previous similar messages [ 1211.359448] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1211.404965] LustreError: 47067:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1211.512557] Lustre: server umount lustre-OST0000 complete [ 1215.969462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.971081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1215.984487] Lustre: Skipped 2 previous similar messages [ 1215.993851] Lustre: Skipped 1 previous similar message [ 1220.374432] LustreError: 47269:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1220.381492] LustreError: 47269:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1220.468650] Lustre: server umount lustre-MDT0000 complete [ 1223.426678] LustreError: 44162:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238598 with bad export cookie 15887405673141277635 [ 1223.427748] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1223.439479] LustreError: 44162:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1223.568376] LustreError: 47473:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1223.576687] LustreError: 47473:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1223.836697] Lustre: server umount lustre-MDT0001 complete [ 1231.104838] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 12:56:45 (1761238605) [ 1236.368547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1236.788228] LustreError: 48192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1236.808808] LustreError: 48192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1236.881596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1240.458263] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1242.083203] LustreError: 48192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1246.179200] LustreError: 48192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1246.609331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1246.909073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1250.174289] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1254.434250] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1257.436036] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1263.121077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1263.361616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1268.297700] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1273.170206] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1286.735475] LustreError: 50412:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1287.457763] Lustre: Modifying parameter general.timeout=1 in log params [ 1289.076159] Lustre: Modifying parameter general.timeout=3 in log params [ 1289.080658] Lustre: Skipped 1 previous similar message [ 1291.205105] Lustre: Modifying parameter general.timeout=6 in log params [ 1291.208320] Lustre: Skipped 2 previous similar messages [ 1295.882139] Lustre: Modifying parameter general.timeout=13 in log params [ 1295.884596] Lustre: Skipped 6 previous similar messages [ 1303.991847] Lustre: Modifying parameter general.timeout=26 in log params [ 1303.994372] Lustre: Skipped 12 previous similar messages [ 1320.269370] Lustre: Modifying parameter general.timeout=53 in log params [ 1320.272568] Lustre: Skipped 26 previous similar messages [ 1352.466912] Lustre: Modifying parameter general.timeout=103 in log params [ 1352.471147] Lustre: Skipped 49 previous similar messages [ 1378.447066] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 12:59:13 (1761238753) [ 1380.131598] Lustre: 57338:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1381.327571] Lustre: 57436:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1383.929974] Lustre: 57632:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1383.937047] Lustre: 57632:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 1 previous similar message [ 1405.155424] Lustre: 59200:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1405.164503] Lustre: 59200:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 2 previous similar messages [ 1405.181546] LustreError: 59200:0:(mgs_handler.c:1126:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1407.971952] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 12:59:42 (1761238782) [ 1415.665234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1415.674483] LustreError: Skipped 1 previous similar message [ 1415.679908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1415.692097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1420.257119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.269450] Lustre: Skipped 1 previous similar message [ 1421.847605] LustreError: 59896:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1421.856807] LustreError: 59896:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1421.917183] Lustre: server umount lustre-MDT0000 complete [ 1424.440331] LustreError: 48172:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238799 with bad export cookie 15887405673141278566 [ 1424.441745] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1424.446110] LustreError: 48172:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1424.689412] Lustre: server umount lustre-MDT0001 complete [ 1437.331112] LustreError: 60298:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1437.334575] LustreError: 60298:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1437.480209] Lustre: server umount lustre-OST0000 complete [ 1445.268103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1449.134061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1451.908885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1455.173207] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1460.359558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1460.576732] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1460.596159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1460.605995] Lustre: Skipped 49 previous similar messages [ 1460.802625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1464.041952] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1471.277291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1471.375419] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1471.394928] Lustre: 61752:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1474.719377] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1478.249178] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1480.843530] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1485.682379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1485.892531] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1490.175981] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1494.431477] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1495.636385] Lustre: 63997:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1501.664621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1501.669881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.677920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1501.682654] Lustre: Skipped 3 previous similar messages [ 1505.787210] LustreError: 64254:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1505.794241] LustreError: 64254:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1505.920210] Lustre: server umount lustre-OST0000 complete [ 1512.416939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1512.417218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1512.421292] LustreError: Skipped 1 previous similar message [ 1512.427554] Lustre: Skipped 2 previous similar messages [ 1514.381623] Lustre: server umount lustre-MDT0000 complete [ 1517.142266] LustreError: 62478:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238892 with bad export cookie 15887405673141281296 [ 1517.142851] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1517.150643] LustreError: 62478:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1517.494820] Lustre: server umount lustre-MDT0001 complete [ 1527.009533] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 1529.801276] Key type lgssc unregistered [ 1530.094399] LNet: 65192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.101671] LNetError: 65192:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.116340] LNet: Removed LNI 192.168.206.113@tcp [ 1530.882662] Key type .llcrypt unregistered [ 1530.884689] Key type ._llcrypt unregistered [ 1547.669199] Key type ._llcrypt registered [ 1547.671687] Key type .llcrypt registered [ 1547.755821] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 1548.318997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1548.335347] alg: No test for adler32 (adler32-zlib) [ 1549.445460] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 1549.740510] LNet: Added LNI 192.168.206.113@tcp [8/256/0/180] [ 1551.423503] Key type lgssc registered [ 1552.467650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1558.897521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1558.909481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1560.287930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1564.131401] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1570.788484] LustreError: 66564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1570.797167] LustreError: 66564:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1570.985690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1571.411310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1574.154244] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1577.759431] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1580.091809] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1586.641359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1587.155375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1591.806739] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1597.195708] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1613.940578] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1620.965697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1620.975025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1620.979073] Lustre: Skipped 2 previous similar messages [ 1621.005893] Lustre: Skipped 2 previous similar messages [ 1625.939282] LustreError: 69296:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1626.000504] Lustre: server umount lustre-MDT0000 complete [ 1626.081639] LustreError: 66564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1626.099894] LustreError: 66564:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1628.820744] LustreError: 66543:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239003 with bad export cookie 15812291910446656961 [ 1628.825204] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1628.835968] LustreError: 66543:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1628.936869] LustreError: 69499:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1628.939901] LustreError: 69499:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1629.179777] Lustre: server umount lustre-MDT0001 complete [ 1642.689153] LustreError: 69700:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1642.691298] LustreError: 69700:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1642.830768] Lustre: server umount lustre-OST0000 complete [ 1648.881343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1651.487414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1653.968116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1656.829094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1662.712688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.912635] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1662.938674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1663.179680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1666.498333] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1674.128538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1674.224757] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1674.244765] Lustre: 71149:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1674.638904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1678.123876] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1683.134239] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1687.633304] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1694.177396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1694.469075] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1694.939226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1700.498743] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1705.917529] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1707.488464] Lustre: 73393:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1709.267472] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1714.149199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1714.149686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1714.150634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1714.170453] Lustre: Skipped 1 previous similar message [ 1719.173184] LustreError: 73652:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1719.181282] LustreError: 73652:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1719.404534] Lustre: server umount lustre-OST0000 complete [ 1724.392541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1724.398948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1724.414685] Lustre: Skipped 1 previous similar message [ 1724.428628] Lustre: Skipped 2 previous similar messages [ 1728.642903] LustreError: 73853:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1728.645950] LustreError: 73853:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1728.718777] Lustre: server umount lustre-MDT0000 complete [ 1729.509360] LustreError: 72230:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1729.533663] LustreError: 72230:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1731.330992] LustreError: 71142:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239106 with bad export cookie 15812291910446657794 [ 1731.335271] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1731.357665] LustreError: 71142:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1731.714718] Lustre: server umount lustre-MDT0001 complete [ 1739.219665] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 1741.114272] Key type lgssc unregistered [ 1741.353932] LNet: 74586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1741.360703] LNetError: 74586:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1741.371692] LNet: Removed LNI 192.168.206.113@tcp [ 1741.883164] Key type .llcrypt unregistered [ 1741.885391] Key type ._llcrypt unregistered [ 1758.621837] Key type ._llcrypt registered [ 1758.623711] Key type .llcrypt registered [ 1758.716175] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 1759.390275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1759.404373] alg: No test for adler32 (adler32-zlib) [ 1760.449735] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 1760.624130] LNet: Added LNI 192.168.206.113@tcp [8/256/0/180] [ 1762.279173] Key type lgssc registered [ 1763.175564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1769.756885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1769.782325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1771.314128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1774.013600] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1779.777017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1780.258053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1783.627755] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1789.051995] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1792.558357] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1799.704621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1800.356405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1805.534585] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1810.802842] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1820.845422] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1821.887070] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1823.473362] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1823.476201] Lustre: Skipped 1 previous similar message [ 1826.050685] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1826.053071] Lustre: Skipped 2 previous similar messages [ 1830.701408] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1830.703845] Lustre: Skipped 5 previous similar messages [ 1839.425640] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1839.428068] Lustre: Skipped 11 previous similar messages [ 1855.547985] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1855.554725] Lustre: Skipped 22 previous similar messages [ 1887.858348] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1887.861401] Lustre: Skipped 44 previous similar messages [ 1900.145207] Lustre: 83717:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1902.522915] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 13:07:56 (1761239276) [ 1903.826548] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1905.502138] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 13:07:59 (1761239279) [ 1908.196847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1908.209558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.221659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1909.217905] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1909.225820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1909.238832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.245054] LustreError: 84012:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1914.338396] LustreError: 77566:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1914.364268] LustreError: 77566:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1914.524338] Lustre: server umount lustre-OST0000 complete [ 1918.432273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1918.441845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.460478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1918.464458] Lustre: Skipped 1 previous similar message [ 1924.577471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1924.580966] Lustre: Skipped 3 previous similar messages [ 1929.697924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1929.700597] Lustre: Skipped 1 previous similar message [ 1932.255191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1932.319605] LustreError: 84217:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1932.325053] LustreError: 84217:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1932.393548] Lustre: server umount lustre-MDT0000 complete [ 1934.817219] LustreError: 75953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1934.827295] LustreError: 75953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1935.228474] LustreError: 75940:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239310 with bad export cookie 3498226893084679746 [ 1935.230476] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1935.235579] LustreError: 75940:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1935.340850] LustreError: 84419:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1935.343574] LustreError: 84419:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1935.493994] Lustre: server umount lustre-MDT0001 complete [ 1943.139247] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 1945.442901] Key type lgssc unregistered [ 1945.703187] LNet: 84950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.709744] LNetError: 84950:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.719824] LNet: Removed LNI 192.168.206.113@tcp [ 1946.454505] Key type .llcrypt unregistered [ 1946.456841] Key type ._llcrypt unregistered [ 1949.967491] Key type ._llcrypt registered [ 1949.969173] Key type .llcrypt registered [ 1950.068601] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1953.014615] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules [ 1953.954258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1953.979499] alg: No test for adler32 (adler32-zlib) [ 1955.097732] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 1955.251555] LNet: Added LNI 192.168.206.113@tcp [8/256/0/180] [ 1956.880621] Key type lgssc registered [ 1957.695740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1958.597232] LustreError: 86115:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1971.409675] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 1978.796542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1980.648846] LustreError: 86115:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1980.851576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.377762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1985.002630] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 1991.983863] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 13:09:26 (1761239366) [ 1992.674325] LustreError: 87123:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1992.690946] LustreError: 87123:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1995.871794] LustreError: 87154:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1995.879572] LustreError: 87154:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1995.884554] LustreError: 87154:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 0, failed: rc = -5 [ 1995.989956] LustreError: 87928:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1996.203644] Lustre: server umount lustre-MDT0000 complete [ 2008.436462] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 2010.769103] Key type lgssc unregistered [ 2011.041701] LNet: 88615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2011.053245] LNetError: 88615:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2011.073928] LNet: Removed LNI 192.168.206.113@tcp [ 2011.717015] Key type .llcrypt unregistered [ 2011.718841] Key type ._llcrypt unregistered [ 2023.628728] Key type ._llcrypt registered [ 2023.630710] Key type .llcrypt registered [ 2023.703869] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 2024.275459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2024.306671] alg: No test for adler32 (adler32-zlib) [ 2025.325958] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 2025.552788] LNet: Added LNI 192.168.206.113@tcp [8/256/0/180] [ 2027.200097] Key type lgssc registered [ 2028.091514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2034.329680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2034.345494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2035.968660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2040.043468] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2045.414029] LustreError: 89793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2045.425049] LustreError: 89793:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2047.586429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2047.980147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2051.342422] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2056.298955] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2059.584640] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2065.820340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2066.313405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.044726] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2075.626658] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2126.068690] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 13:11:39 (1761239499) [ 2127.760266] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2129.335775] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 13:11:43 (1761239503) [ 2135.008617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2135.018305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2135.041538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2137.068666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2137.071713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2138.908153] LustreError: 92703:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2138.965280] Lustre: server umount lustre-MDT0000 complete [ 2142.101144] LustreError: 89774:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239517 with bad export cookie 3512439976620586893 [ 2142.106566] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2142.116682] LustreError: 89774:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2142.175844] LustreError: 89787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2142.180642] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.213820] Lustre: Skipped 1 previous similar message [ 2142.219883] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2142.222749] Lustre: Skipped 2 previous similar messages [ 2142.242987] LustreError: 92904:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2142.246643] LustreError: 92904:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2142.413209] Lustre: server umount lustre-MDT0001 complete [ 2151.242559] LustreError: 93106:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2151.261910] LustreError: 93106:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2151.599088] Lustre: server umount lustre-OST0000 complete [ 2160.148394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2160.429092] LustreError: 93735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2160.550444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.408682] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2165.734260] LustreError: 93735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.988169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2170.437135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2173.002239] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2176.888143] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2179.748727] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2184.547840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2189.026626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2189.075236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2189.204391] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2189.208661] LustreError: 93722:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2189.231165] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2189.236597] LustreError: 95598:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -98 [ 2189.239853] LustreError: 95598:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2189.243829] LustreError: 95598:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2189.332638] Lustre: server umount lustre-OST0000 complete [ 2189.334621] LustreError: 95598:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2194.882582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2194.980322] LustreError: 96069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.008048] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2195.012269] LustreError: 93723:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2195.018824] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2195.023919] LustreError: 96061:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -98 [ 2195.027348] LustreError: 96061:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2195.029604] LustreError: 96061:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2195.100465] Lustre: server umount lustre-OST0000 complete [ 2195.102725] LustreError: 96061:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2196.728286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2200.845175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2200.989667] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2200.994868] Lustre: Found index 0 for lustre-OST0000, updating log [ 2201.003439] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2201.064616] Lustre: lustre-OST0000: new disk, initializing [ 2201.069606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2201.247708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2205.507325] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2208.191146] Lustre: Failing over lustre-OST0000 [ 2208.243090] LustreError: 97147:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2208.247434] LustreError: 97147:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2208.361960] Lustre: server umount lustre-OST0000 complete [ 2211.299862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2211.306788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2211.316109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2216.930951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2216.935879] Lustre: Skipped 2 previous similar messages [ 2220.019121] LustreError: 97348:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2220.024016] LustreError: 97348:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2220.071527] Lustre: server umount lustre-MDT0000 complete [ 2222.049818] LustreError: 94535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.063599] LustreError: 94535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2222.585052] LustreError: 93714:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239597 with bad export cookie 3512439976620589637 [ 2222.590124] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2222.592338] LustreError: 93714:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2222.865535] Lustre: server umount lustre-MDT0001 complete [ 2231.040895] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 13:13:25 (1761239605) [ 2243.402904] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 2250.437271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2250.720493] LustreError: 98918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.825925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2253.806574] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2261.143290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2265.315774] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2267.883597] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2274.646988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2275.311250] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2276.322789] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2276.348585] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2276.357763] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2276.364401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2280.099262] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2286.670127] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2286.735580] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2286.779837] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2286.873340] Lustre: lustre-OST0001: new disk, initializing [ 2286.879171] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2287.171384] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2287.173816] Lustre: Skipped 2 previous similar messages [ 2287.689406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2287.699549] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2287.796992] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2291.574526] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2298.461743] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2304.228994] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2313.187887] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2313.191872] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.212703] Lustre: Skipped 1 previous similar message [ 2313.215932] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2319.274519] LustreError: 102159:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2319.277255] LustreError: 102159:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2319.368533] Lustre: server umount lustre-MDT0001 complete [ 2321.722398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2323.940846] LustreError: 98917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.955440] LustreError: 98917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2327.161494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2327.250678] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2327.253384] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2327.303228] Lustre: 98906:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2329.140659] LustreError: 102771:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2329.148368] LustreError: 102771:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2329.158133] LustreError: 98909:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2329.223465] LustreError: 102776:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.206.113@tcp: cfg command failed: rc = -17 [ 2329.230586] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2329.230586] [ 2331.276257] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2333.173909] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2340.233947] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 13:15:14 (1761239714) [ 2354.199045] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 2368.366918] Lustre: 104771:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2388.858164] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2391.795836] Lustre: 105905:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2397.044198] Lustre: 100819:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 506 < left 1007, rollback = 2 [ 2397.052673] Lustre: 100819:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 4/16/6, destroy: 0/0/0 [ 2397.058970] Lustre: 100819:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 8/1007/0 [ 2397.063567] Lustre: 100819:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 2397.068136] Lustre: 100819:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 12/236/4, delete: 0/0/0 [ 2397.071844] Lustre: 100819:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0 [ 2430.555979] Lustre: 98914:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 509 < left 993, rollback = 2 [ 2430.561094] Lustre: 98914:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 8 previous similar messages [ 2430.564630] Lustre: 98914:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/4, destroy: 0/0/0 [ 2430.568305] Lustre: 98914:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2430.572055] Lustre: 98914:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 7/993/0 [ 2430.574892] Lustre: 98914:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2430.578898] Lustre: 98914:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 5/58/0, punch: 0/0/0, quota 1/3/0 [ 2430.583510] Lustre: 98914:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2430.587360] Lustre: 98914:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 10/203/3, delete: 0/0/0 [ 2430.591272] Lustre: 98914:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2430.594493] Lustre: 98914:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0 [ 2430.598749] Lustre: 98914:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2476.004517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2476.013204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.022937] Lustre: Skipped 2 previous similar messages [ 2476.030062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2476.035703] Lustre: Skipped 7 previous similar messages [ 2479.958666] LustreError: 106363:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2479.970227] LustreError: 106363:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2480.080706] Lustre: server umount lustre-MDT0000 complete [ 2481.642601] LustreError: 98917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.658637] LustreError: 98917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2483.306775] LustreError: 98898:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239858 with bad export cookie 3512439976620590568 [ 2483.312075] LustreError: 98898:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2483.314365] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2483.679877] Lustre: server umount lustre-MDT0001 complete [ 2497.429435] Lustre: server umount lustre-OST0000 complete [ 2510.344914] Lustre: server umount lustre-OST0001 complete [ 2515.687044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2522.177866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2522.740772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2528.144818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2535.256371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2535.955506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2550.126634] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 2557.391168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2557.401966] Lustre: lustre-MDT0000: reset Object Index mappings [ 2557.882797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2557.886115] Lustre: Skipped 1 previous similar message [ 2560.577469] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2567.516162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2567.526142] Lustre: lustre-MDT0001: reset Object Index mappings [ 2571.580917] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2574.211067] Lustre: 110963:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2579.194416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2582.773785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 2583.761857] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2589.491085] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2590.991350] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2592.059651] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 2592.064258] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2594.044756] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2600.685557] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2603.543498] Lustre: 112818:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2642.843522] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:20:17 (1761240017) [ 2674.148243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2674.151039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2674.168907] Lustre: Skipped 5 previous similar messages [ 2674.170534] Lustre: Skipped 4 previous similar messages [ 2677.765046] LustreError: 113170:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2677.770955] LustreError: 113170:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2677.881614] Lustre: server umount lustre-MDT0000 complete [ 2679.264750] LustreError: 110588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.283754] LustreError: 110588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2680.489150] LustreError: 112142:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761240055 with bad export cookie 3512439976620637657 [ 2680.499245] LustreError: 112142:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2680.501193] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2686.869543] Lustre: server umount lustre-MDT0001 complete [ 2695.356704] Lustre: server umount lustre-OST0000 complete [ 2704.659282] Lustre: server umount lustre-OST0001 complete [ 2710.069766] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 2718.164786] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 2725.924661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2731.020183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2735.191994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2739.757686] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2744.651623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2744.710280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2744.853191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2744.894291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2744.938657] Lustre: lustre-MDT0000: new disk, initializing [ 2745.033382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2745.045127] Lustre: Skipped 3 previous similar messages [ 2745.066635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2748.348628] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2754.287855] LustreError: 116536:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2754.290991] LustreError: 116536:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2754.451633] Lustre: server umount lustre-MDT0000 complete [ 2756.339964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2762.603362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2762.671708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.872797] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2762.882812] Lustre: 117054:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2762.895655] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2766.908918] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2771.403867] Lustre: server umount lustre-MDT0000 complete [ 2778.179343] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:22:32 (1761240152) [ 2779.826164] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2781.324360] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:22:35 (1761240155) [ 2795.414321] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 2802.402686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2805.449074] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2812.350654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2812.427903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2812.520617] Lustre: 119071:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2812.574638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2812.577259] Lustre: Skipped 1 previous similar message [ 2812.645436] Lustre: lustre-MDT0001: new disk, initializing [ 2812.803378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2812.815523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2812.926644] Lustre: 119812:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 2812.943919] Lustre: 119812:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 8 previous similar messages [ 2812.951039] Lustre: 119812:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 2812.961464] Lustre: 119812:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2812.980535] Lustre: 119812:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 2812.995193] Lustre: 119812:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2813.004976] Lustre: 119812:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 2813.010849] Lustre: 119812:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2813.028847] Lustre: 119812:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 2813.032009] Lustre: 119812:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2813.040829] Lustre: 119812:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 2813.051067] Lustre: 119812:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2815.945269] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2820.540042] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2825.362228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2825.433592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2825.592048] Lustre: lustre-OST0000: new disk, initializing [ 2825.596745] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2827.297446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2827.310586] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2827.395509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2829.951926] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2838.547743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2838.592561] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2838.699411] Lustre: lustre-OST0001: new disk, initializing [ 2838.702473] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2840.452972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2840.460206] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2840.515515] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2842.516148] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 2850.235892] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2852.711609] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2857.932503] Lustre: 119078:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 504 < left 1021, rollback = 2 [ 2857.942852] Lustre: 119078:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 2857.946878] Lustre: 119078:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 4/16/8, destroy: 0/0/0 [ 2857.955024] Lustre: 119078:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2857.960566] Lustre: 119078:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 9/1021/0 [ 2857.964401] Lustre: 119078:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2857.967309] Lustre: 119078:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/153/0, punch: 0/0/0, quota 1/3/0 [ 2857.972442] Lustre: 119078:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2857.979932] Lustre: 119078:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 13/232/4, delete: 0/0/0 [ 2857.983164] Lustre: 119078:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2857.985602] Lustre: 119078:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 5/5/0, ref_del: 0/0/0 [ 2857.989149] Lustre: 119078:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3206.818550] Lustre: 127872:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3206.832674] Lustre: 127872:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 8 previous similar messages [ 3206.844750] Lustre: 127872:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3206.851558] Lustre: 127872:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 3206.859033] Lustre: 127872:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3206.872711] Lustre: 127872:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 3206.882600] Lustre: 127872:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3206.887888] Lustre: 127872:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 3206.892313] Lustre: 127872:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3206.896730] Lustre: 127872:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 3206.900695] Lustre: 127872:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3206.911663] Lustre: 127872:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 3458.316801] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:33:51 (1761240831) [ 3473.377770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3473.384329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3473.391379] Lustre: Skipped 3 previous similar messages [ 3473.403582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3473.408252] Lustre: Skipped 6 previous similar messages [ 3474.775105] LustreError: 128100:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3474.778391] LustreError: 128100:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3474.927384] Lustre: server umount lustre-MDT0000 complete [ 3475.429686] LustreError: 122288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3475.447967] LustreError: 122288:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3478.124564] LustreError: 122293:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761240853 with bad export cookie 3512439976620701476 [ 3478.127605] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3478.149565] LustreError: 122293:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3480.546672] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3480.571775] Lustre: Skipped 4 previous similar messages [ 3483.552863] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3483.562095] Lustre: Skipped 3 previous similar messages [ 3484.552886] Lustre: server umount lustre-MDT0001 complete [ 3494.348082] Lustre: server umount lustre-OST0000 complete [ 3512.287308] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3512.558651] Lustre: server umount lustre-OST0001 complete [ 3518.212180] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 3525.521745] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 3534.588632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3541.913314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3548.860985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3554.830952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3561.767164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3561.855577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3562.024413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3562.050044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3562.103586] Lustre: lustre-MDT0000: new disk, initializing [ 3562.215939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3562.222928] Lustre: Skipped 5 previous similar messages [ 3562.251611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3565.793259] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 3576.739697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3576.858398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3577.032508] Lustre: 130896:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3577.110831] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3577.124093] Lustre: Skipped 1 previous similar message [ 3577.278859] Lustre: lustre-MDT0001: new disk, initializing [ 3577.478623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3577.491588] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3577.614378] Lustre: 131693:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 3577.632181] Lustre: 131693:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 255 previous similar messages [ 3577.656079] Lustre: 131693:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 3577.674056] Lustre: 131693:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 3577.690800] Lustre: 131693:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 3577.702057] Lustre: 131693:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 3577.724209] Lustre: 131693:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 3577.762611] Lustre: 131693:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 3577.790821] Lustre: 131693:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 3577.804482] Lustre: 131693:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 3577.819714] Lustre: 131693:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 3577.838441] Lustre: 131693:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 3582.151885] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 3590.675422] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3595.022573] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3602.892677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3603.018040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3603.380257] Lustre: lustre-OST0000: new disk, initializing [ 3603.390866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3605.288754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3605.297069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3605.443689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3608.832330] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 3615.812938] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3625.757376] Lustre: lustre-MDD0000: changelog on [ 3628.345685] Lustre: lustre-MDD0001: changelog on [ 3631.852980] Lustre: 130903:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 539, rollback = 2 [ 3631.875654] Lustre: 130903:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 3631.883251] Lustre: 130903:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/1, destroy: 0/0/0 [ 3631.892025] Lustre: 130903:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3631.902425] Lustre: 130903:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 9/539/0 [ 3631.910806] Lustre: 130903:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3631.923255] Lustre: 130903:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 10/93/1, punch: 0/0/0, quota 1/3/0 [ 3631.933289] Lustre: 130903:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3631.948156] Lustre: 130903:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 4/67/1, delete: 0/0/0 [ 3631.959279] Lustre: 130903:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3631.973585] Lustre: 130903:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3631.985956] Lustre: 130903:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3663.862449] Lustre: 130901:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 515 < left 525, rollback = 2 [ 3663.865503] Lustre: 130901:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 6812 previous similar messages [ 3663.892911] Lustre: 130902:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0 [ 3663.897771] Lustre: 130902:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 6818 previous similar messages [ 3663.903990] Lustre: 130902:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 3663.912735] Lustre: 130902:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 6818 previous similar messages [ 3663.928581] Lustre: 130902:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/39/0, punch: 0/0/0, quota 1/3/0 [ 3663.936664] Lustre: 130902:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 6821 previous similar messages [ 3663.959140] Lustre: 130902:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 3/50/0, delete: 0/0/0 [ 3663.962928] Lustre: 130902:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 6824 previous similar messages [ 3663.981370] Lustre: 130902:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3663.985279] Lustre: 130902:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 6827 previous similar messages [ 3737.041979] Lustre: 130903:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 508 < left 525, rollback = 2 [ 3737.062811] Lustre: 130903:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 6686 previous similar messages [ 3737.073990] Lustre: 130903:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/6, destroy: 0/0/0 [ 3737.082807] Lustre: 130903:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 6680 previous similar messages [ 3737.090743] Lustre: 130903:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 3737.101406] Lustre: 130903:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 6680 previous similar messages [ 3737.115421] Lustre: 130903:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/39/1, punch: 0/0/0, quota 1/3/0 [ 3737.124263] Lustre: 130903:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 6677 previous similar messages [ 3737.130681] Lustre: 130903:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 3/50/1, delete: 0/0/0 [ 3737.133827] Lustre: 130903:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 6674 previous similar messages [ 3737.137734] Lustre: 130903:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3737.140854] Lustre: 130903:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 6671 previous similar messages [ 3865.049473] Lustre: 130903:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 525, rollback = 2 [ 3865.055464] Lustre: 130903:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 20633 previous similar messages [ 3865.088188] Lustre: 132032:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0 [ 3865.097911] Lustre: 132032:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 20639 previous similar messages [ 3865.111179] Lustre: 132032:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 3865.117578] Lustre: 132032:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 20639 previous similar messages [ 3865.129787] Lustre: 132032:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/39/0, punch: 0/0/0, quota 1/3/0 [ 3865.138139] Lustre: 132032:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 20639 previous similar messages [ 3865.148263] Lustre: 132032:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 3/50/0, delete: 0/0/0 [ 3865.152822] Lustre: 132032:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 20639 previous similar messages [ 3865.157276] Lustre: 132032:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3865.161761] Lustre: 132032:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 20639 previous similar messages [ 4123.984584] Lustre: 130903:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 508 < left 525, rollback = 2 [ 4123.998817] Lustre: 130903:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 33365 previous similar messages [ 4124.008071] Lustre: 130903:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/6, destroy: 0/0/0 [ 4124.020062] Lustre: 130903:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 33359 previous similar messages [ 4124.030570] Lustre: 130903:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 4124.034327] Lustre: 130903:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 33359 previous similar messages [ 4124.044980] Lustre: 130903:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/39/1, punch: 0/0/0, quota 1/3/0 [ 4124.064532] Lustre: 130903:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 33359 previous similar messages [ 4124.071410] Lustre: 130903:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 3/50/1, delete: 0/0/0 [ 4124.084723] Lustre: 130903:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 33359 previous similar messages [ 4124.088954] Lustre: 130903:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4124.092340] Lustre: 130903:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 33359 previous similar messages [ 4636.079364] Lustre: 130901:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 525, rollback = 2 [ 4636.098511] Lustre: 130901:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 67499 previous similar messages [ 4636.103676] Lustre: 130901:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/1, destroy: 0/0/0 [ 4636.109196] Lustre: 130901:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 67499 previous similar messages [ 4636.122860] Lustre: 130901:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 4636.135688] Lustre: 130901:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 67499 previous similar messages [ 4636.151842] Lustre: 130901:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/39/1, punch: 0/0/0, quota 1/3/0 [ 4636.160456] Lustre: 130901:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 67499 previous similar messages [ 4636.175553] Lustre: 130901:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 3/50/1, delete: 0/0/0 [ 4636.188439] Lustre: 130901:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 67499 previous similar messages [ 4636.203611] Lustre: 130901:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4636.222690] Lustre: 130901:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 67499 previous similar messages [ 5272.704890] Lustre: 130901:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 525, rollback = 2 [ 5272.709425] Lustre: 130901:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 80999 previous similar messages [ 5272.719573] Lustre: 130901:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/1, destroy: 0/0/0 [ 5272.725144] Lustre: 130901:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 80999 previous similar messages [ 5272.729014] Lustre: 130901:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 5272.733273] Lustre: 130901:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 80999 previous similar messages [ 5272.737179] Lustre: 130901:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/39/1, punch: 0/0/0, quota 1/3/0 [ 5272.741382] Lustre: 130901:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 80999 previous similar messages [ 5272.746179] Lustre: 130901:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 3/50/1, delete: 0/0/0 [ 5272.749715] Lustre: 130901:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 80999 previous similar messages [ 5272.755867] Lustre: 130901:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 5272.762399] Lustre: 130901:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 80999 previous similar messages [ 5485.213616] Lustre: lustre-MDD0001: changelog off [ 5486.679537] Lustre: 130902:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5487.447249] Lustre: lustre-MDD0000: changelog off [ 5494.755596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5494.756124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5494.759253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5494.759346] LustreError: Skipped 1 previous similar message [ 5494.771661] Lustre: Skipped 1 previous similar message [ 5497.222170] LustreError: 140735:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5497.228932] LustreError: 140735:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5497.358313] Lustre: server umount lustre-OST0000 complete [ 5499.874700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5499.881960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5499.884364] Lustre: Skipped 1 previous similar message [ 5499.888600] Lustre: Skipped 1 previous similar message [ 5505.000036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5505.003784] Lustre: Skipped 2 previous similar messages [ 5514.207133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5514.210572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5514.220513] Lustre: Skipped 3 previous similar messages [ 5514.570985] Lustre: server umount lustre-MDT0000 complete [ 5517.943647] LustreError: 130886:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761242893 with bad export cookie 3512439976621473940 [ 5517.951348] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5518.298097] Lustre: server umount lustre-MDT0001 complete [ 5528.766240] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 5531.562731] Key type lgssc unregistered [ 5531.928785] LNet: 141672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5531.937495] LNetError: 141672:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5531.959560] LNet: Removed LNI 192.168.206.113@tcp [ 5532.709357] Key type .llcrypt unregistered [ 5532.715293] Key type ._llcrypt unregistered [ 5548.138479] Key type ._llcrypt registered [ 5548.140449] Key type .llcrypt registered [ 5548.257350] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 5562.276218] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 5563.191505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5563.207887] alg: No test for adler32 (adler32-zlib) [ 5564.252212] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 5564.436462] LNet: Added LNI 192.168.206.113@tcp [8/256/0/180] [ 5566.055172] Key type lgssc registered [ 5566.862769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5574.107720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5579.075153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5583.682294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5588.529433] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5593.376891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5593.409504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5593.419800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5594.623487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5594.652406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5594.703666] Lustre: lustre-MDT0000: new disk, initializing [ 5594.758864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5594.771849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5597.401666] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 5606.439260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5606.509931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5606.589197] Lustre: 144847:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5606.655678] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5606.658654] Lustre: Skipped 1 previous similar message [ 5606.750545] Lustre: lustre-MDT0001: new disk, initializing [ 5606.830516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5606.881220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5606.891964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5606.971077] Lustre: 145645:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 5606.983313] Lustre: 145645:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5607.006307] Lustre: 145645:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 5607.010684] Lustre: 145645:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 5607.015159] Lustre: 145645:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5607.018876] Lustre: 145645:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5610.441630] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 5616.633808] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5619.842790] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5625.803519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5625.893841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5626.202250] Lustre: lustre-OST0000: new disk, initializing [ 5626.205416] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5626.323357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5630.856374] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 5632.578914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5632.594765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5632.721874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5637.448732] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5642.945449] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_module obdecho/obdecho [ 5644.692389] LustreError: 147471:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5644.714562] LustreError: 147471:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 5646.667789] LustreError: 147568:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 5658.081304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5658.085996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5658.094835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5658.608071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5658.615543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5658.622829] Lustre: Skipped 1 previous similar message [ 5658.628262] Lustre: Skipped 1 previous similar message [ 5663.130515] LustreError: 147908:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5663.229336] Lustre: server umount lustre-MDT0000 complete [ 5666.382527] LustreError: 144838:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243041 with bad export cookie 5105856801741986954 [ 5666.386664] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5666.390821] LustreError: 144838:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5666.415115] LustreError: 148110:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000826799b0 x1846797076045696/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5666.439431] LustreError: 148110:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x20000000a:0x0:0x0]: rc = -5 [ 5666.444408] LustreError: 148110:0:(llog_cat.c:443:llog_cat_close()) lustre-MDT0000-osp-MDT0001: failure destroying log during cleanup: rc = -5 [ 5666.586990] LustreError: 148110:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5666.594214] LustreError: 148110:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5666.835485] Lustre: server umount lustre-MDT0001 complete [ 5680.306864] LustreError: 148310:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5680.309887] LustreError: 148310:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5680.438341] Lustre: server umount lustre-OST0000 complete [ 5696.689293] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 5703.777777] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 5711.217419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5716.133314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5722.206553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5728.769173] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5736.705881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5736.780800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5737.015660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5737.063623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5737.159509] Lustre: lustre-MDT0000: new disk, initializing [ 5737.249753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5737.276072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5741.514354] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 5750.921720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5750.985662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5751.047375] Lustre: 151257:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5751.112193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5751.114856] Lustre: Skipped 1 previous similar message [ 5751.178740] Lustre: lustre-MDT0001: new disk, initializing [ 5751.305063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5751.370086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5751.383262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5751.462139] Lustre: 152058:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 5751.474063] Lustre: 152058:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 5751.483839] Lustre: 152058:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5751.493376] Lustre: 152058:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5751.497891] Lustre: 152058:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 5751.507450] Lustre: 152058:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5751.509746] Lustre: 152058:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 5751.516689] Lustre: 152058:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5751.520065] Lustre: 152058:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5751.549285] Lustre: 152058:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5751.555859] Lustre: 152058:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5751.563212] Lustre: 152058:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5754.488818] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 5760.508857] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5763.228357] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5768.283795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5768.331126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5768.543149] Lustre: lustre-OST0000: new disk, initializing [ 5768.547485] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5768.648380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5772.625984] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 5777.949690] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5777.971558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5777.979994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5778.106741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5780.879076] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5781.037863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5783.713053] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5783.867766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5788.646179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5788.646557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5788.651478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5788.653827] Lustre: Skipped 1 previous similar message [ 5788.658946] Lustre: Skipped 1 previous similar message [ 5793.761782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5798.887297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5798.890844] Lustre: Skipped 2 previous similar messages [ 5799.392279] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5799.420487] LustreError: 154200:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5799.424126] LustreError: 154200:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5799.513371] Lustre: server umount lustre-OST0000 complete [ 5802.468224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5802.472195] LustreError: Skipped 1 previous similar message [ 5802.475320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5809.120917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5809.123731] Lustre: Skipped 4 previous similar messages [ 5816.289193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5816.380739] LustreError: 154403:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5816.384477] LustreError: 154403:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5816.460972] Lustre: server umount lustre-MDT0000 complete [ 5819.360146] LustreError: 151270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5819.382394] LustreError: 151270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5819.529187] LustreError: 153009:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243194 with bad export cookie 5105856801741987955 [ 5819.532253] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5819.538942] LustreError: 153009:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5819.962968] Lustre: server umount lustre-MDT0001 complete [ 5823.084332] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:13:17 (1761243197) [ 5824.344853] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5825.947729] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:13:20 (1761243200) [ 5832.225426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5832.660716] LustreError: 155278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5832.709180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5835.624727] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 5837.796104] LustreError: 155278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5842.913268] LustreError: 155278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.223966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5847.690166] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 5852.687943] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5855.930957] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5863.243937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5867.331925] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 5871.045584] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5881.976932] Lustre: 155273:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 504 < left 1021, rollback = 2 [ 5881.988684] Lustre: 155273:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 5881.992723] Lustre: 155273:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 4/16/8, destroy: 0/0/0 [ 5881.997143] Lustre: 155273:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5882.001930] Lustre: 155273:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 9/1021/0 [ 5882.005813] Lustre: 155273:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5882.011940] Lustre: 155273:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/153/0, punch: 0/0/0, quota 1/3/0 [ 5882.016121] Lustre: 155273:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5882.021957] Lustre: 155273:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 13/232/4, delete: 0/0/0 [ 5882.025630] Lustre: 155273:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5882.029848] Lustre: 155273:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 5/5/0, ref_del: 0/0/0 [ 5882.034317] Lustre: 155273:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5889.509084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5889.514701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5889.515395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5889.515399] Lustre: Skipped 2 previous similar messages [ 5889.526969] Lustre: Skipped 3 previous similar messages [ 5901.279261] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5901.345233] LustreError: 157546:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5901.353552] LustreError: 157546:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5901.412987] Lustre: server umount lustre-MDT0000 complete [ 5902.109306] LustreError: 155272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5904.360427] LustreError: 155257:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243279 with bad export cookie 5105856801741988879 [ 5904.363671] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5904.382870] LustreError: 155257:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5904.864677] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5910.997850] Lustre: server umount lustre-MDT0001 complete [ 5918.357920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5931.811742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5932.096691] LustreError: 158823:0:(llog.c:1616:llog_backup()) MGC192.168.206.113@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5932.112456] Lustre: 158823:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.206.113@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5934.561731] LustreError: 158831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5934.579690] LustreError: 158831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5954.321647] LustreError: 158831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5954.340132] LustreError: 158831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 5989.133196] LustreError: 158830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5989.159288] LustreError: 158830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 5992.927630] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5992.929993] Lustre: 158823:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.206.113@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6021.599290] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6021.602161] Lustre: 155269:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.206.113@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6033.909845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6033.919785] Lustre: Skipped 2 previous similar messages [ 6037.775322] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6039.021200] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6042.082274] LustreError: 155269:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6046.298877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6050.271955] LustreError: 143058:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006d71b09b x1846797076136960/t0(0) o250->MGC192.168.206.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6053.766115] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6055.922077] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6057.967246] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6059.376592] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6062.324698] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6063.611984] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6081.517872] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6081.518583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6081.523371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6081.523382] Lustre: Skipped 13 previous similar messages [ 6081.535099] Lustre: Skipped 1 previous similar message [ 6083.943383] LustreError: 160328:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6083.957683] LustreError: 160328:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6084.048095] Lustre: server umount lustre-MDT0000 complete [ 6086.627137] LustreError: 156888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6086.635039] LustreError: 156888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 47 previous similar messages [ 6087.025823] LustreError: 155259:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243462 with bad export cookie 5105856801741995354 [ 6087.031397] LustreError: 155259:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6087.031982] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6087.329626] Lustre: server umount lustre-MDT0001 complete [ 6100.367137] Lustre: server umount lustre-OST0000 complete [ 6115.815965] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 6124.011994] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 6131.594185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6137.512576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6143.907339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6150.192584] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6156.406563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6156.486149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6156.741679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6156.802687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6156.901405] Lustre: lustre-MDT0000: new disk, initializing [ 6156.986936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6156.990068] Lustre: Skipped 1 previous similar message [ 6157.005439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6160.934143] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6170.650674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6170.704583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6170.758547] Lustre: 163679:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6170.817505] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6170.821443] Lustre: Skipped 2 previous similar messages [ 6171.129517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6171.258586] Lustre: 164480:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6171.269787] Lustre: 164480:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 3 previous similar messages [ 6171.276448] Lustre: 164480:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6171.280818] Lustre: 164480:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 6171.287502] Lustre: 164480:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6171.298655] Lustre: 164480:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 6171.332187] Lustre: 164480:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6171.342832] Lustre: 164480:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 6171.359993] Lustre: 164480:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6171.377228] Lustre: 164480:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 6171.398985] Lustre: 164480:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6171.403937] Lustre: 164480:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 6175.219698] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6181.843904] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6185.091720] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6192.170898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6192.229425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6192.443837] Lustre: lustre-OST0000: new disk, initializing [ 6192.447518] Lustre: Skipped 1 previous similar message [ 6192.452400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6192.455536] Lustre: Skipped 2 previous similar messages [ 6197.543379] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6201.937035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6201.949772] Lustre: Skipped 1 previous similar message [ 6201.954143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6202.038373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6204.680115] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6208.932723] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6209.162502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6213.375971] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6213.555739] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6217.695985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6217.698655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6217.742090] Lustre: Skipped 2 previous similar messages [ 6221.620069] LustreError: 166619:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6221.631465] LustreError: 166619:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6221.842557] Lustre: server umount lustre-OST0000 complete [ 6231.953146] Lustre: server umount lustre-MDT0000 complete [ 6233.057384] LustreError: 164930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.086573] LustreError: 164930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6236.071678] LustreError: 166823:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243611 with bad export cookie 5105856801742012175 [ 6236.081077] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6236.097392] LustreError: 166823:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6236.475127] Lustre: server umount lustre-MDT0001 complete [ 6245.630753] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:20:19 (1761243619) [ 6252.702485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6257.256405] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6265.316207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6269.429971] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6274.442973] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6278.480167] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6286.977174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6287.521676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6287.532409] Lustre: Skipped 4 previous similar messages [ 6293.636914] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6299.018860] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6308.635722] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:21:22 (1761243682) [ 6314.467436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6314.471345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6314.473744] LustreError: Skipped 2 previous similar messages [ 6314.475024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6314.485751] Lustre: Skipped 3 previous similar messages [ 6314.493975] Lustre: Skipped 5 previous similar messages [ 6318.148567] LustreError: 170138:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6318.162536] LustreError: 170138:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6318.282973] Lustre: server umount lustre-OST0000 complete [ 6327.588072] Lustre: server umount lustre-MDT0000 complete [ 6330.595338] LustreError: 167676:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243705 with bad export cookie 5105856801742013099 [ 6330.601048] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6330.601455] LustreError: 167676:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6330.911317] Lustre: server umount lustre-MDT0001 complete [ 6339.959703] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 6342.311795] Key type lgssc unregistered [ 6342.601735] LNet: 171073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6342.606106] LNetError: 171073:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6342.618055] LNet: Removed LNI 192.168.206.113@tcp [ 6343.377377] Key type .llcrypt unregistered [ 6343.382748] Key type ._llcrypt unregistered [ 6360.877448] Key type ._llcrypt registered [ 6360.878834] Key type .llcrypt registered [ 6361.019794] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 6362.206627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6362.222757] alg: No test for adler32 (adler32-zlib) [ 6363.256256] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 6363.469822] LNet: Added LNI 192.168.206.113@tcp [8/256/0/180] [ 6365.183241] Key type lgssc registered [ 6366.095696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6373.368762] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6373.401771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6388.897668] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6394.015233] LustreError: 172296:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.113@tcp: failed processing log, type 4: rc = -110 [ 6399.135846] LustreError: 172296:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6399.144885] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6399.158501] LustreError: MGC192.168.206.113@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6399.179230] LustreError: 172296:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6399.198243] LustreError: 172296:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -22 [ 6399.215126] LustreError: 172296:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 6399.221960] LustreError: 172296:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 6399.316886] Lustre: server umount lustre-OST0000 complete [ 6399.319544] LustreError: 172296:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6407.697778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6409.325806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6413.717626] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6414.832031] LustreError: 172798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6414.849797] LustreError: 172798:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6418.922389] LustreError: 172798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6421.831399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6422.157295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6425.747155] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6431.015196] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6434.434752] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6440.998085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6441.283501] LustreError: 174395:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6441.292523] Lustre: 174395:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.206.113@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6441.487120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6446.705242] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6451.461439] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6457.833103] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6457.840544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6457.841210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6457.842900] Lustre: Skipped 2 previous similar messages [ 6459.428424] LustreError: 175046:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6459.436408] LustreError: 175046:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6459.530660] Lustre: server umount lustre-MDT0000 complete [ 6462.958313] LustreError: 172798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6463.245226] LustreError: 174424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243838 with bad export cookie 13050418595860496303 [ 6463.258248] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6463.375983] LustreError: 175248:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6463.388410] LustreError: 175248:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6463.719126] Lustre: server umount lustre-MDT0001 complete [ 6477.461775] LustreError: 175448:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6477.465239] LustreError: 175448:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6477.674897] Lustre: server umount lustre-OST0000 complete [ 6486.543618] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 6495.271554] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 6503.535031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6509.667819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6515.225774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6521.151474] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6527.678341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6527.769372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6528.088851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6528.150150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6528.264899] Lustre: lustre-MDT0000: new disk, initializing [ 6528.412286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6528.441879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6532.624621] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6544.097927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6544.160102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6544.268657] Lustre: 177789:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6544.345088] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6544.347949] Lustre: Skipped 1 previous similar message [ 6544.411537] Lustre: lustre-MDT0001: new disk, initializing [ 6544.502442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6544.540024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6544.556504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6544.613090] Lustre: 178586:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6544.631449] Lustre: 178586:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6544.636415] Lustre: 178586:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6544.645952] Lustre: 178586:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6544.658332] Lustre: 178586:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6544.663304] Lustre: 178586:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6548.262672] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6554.764878] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6557.849632] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6564.134272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6564.200248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6564.448340] Lustre: lustre-OST0000: new disk, initializing [ 6564.455440] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6564.549050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6569.866533] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6571.595133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6571.605769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6571.786052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6577.471473] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6582.456090] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6582.677297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6587.076601] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6587.289481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6592.482563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6592.488185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6592.523227] LustreError: Skipped 1 previous similar message [ 6592.527128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6592.527143] Lustre: Skipped 2 previous similar messages [ 6592.547103] Lustre: Skipped 1 previous similar message [ 6595.227117] LustreError: 180732:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6595.237349] LustreError: 180732:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6595.376927] Lustre: server umount lustre-OST0000 complete [ 6600.673904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6600.682054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6600.696253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6600.700206] Lustre: Skipped 1 previous similar message [ 6602.722722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6602.726329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6602.767837] Lustre: Skipped 1 previous similar message [ 6604.874781] LustreError: 180934:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6604.889521] LustreError: 180934:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6605.012281] Lustre: server umount lustre-MDT0000 complete [ 6607.842744] LustreError: 177795:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6607.857450] LustreError: 177795:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6608.408458] LustreError: 177780:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243983 with bad export cookie 13050418595860497010 [ 6608.414590] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6608.429696] LustreError: 177780:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6608.910886] Lustre: server umount lustre-MDT0001 complete [ 6617.089411] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 14:26:31 (1761243991) [ 6630.659780] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 6633.099561] Key type lgssc unregistered [ 6633.327717] LNet: 182318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6633.343286] LNetError: 182318:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6633.360971] LNet: Removed LNI 192.168.206.113@tcp [ 6634.092709] Key type .llcrypt unregistered [ 6634.094327] Key type ._llcrypt unregistered [ 6650.816647] Key type ._llcrypt registered [ 6650.818183] Key type .llcrypt registered [ 6650.984358] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 6652.371079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6652.402957] alg: No test for adler32 (adler32-zlib) [ 6653.463571] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 6653.665935] LNet: Added LNI 192.168.206.113@tcp [8/256/0/180] [ 6655.351186] Key type lgssc registered [ 6656.479217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6664.183847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6664.216875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6665.742187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6670.399473] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6674.919835] LustreError: 183517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6674.950271] LustreError: 183517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6679.059522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6679.375427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6684.433952] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6690.091687] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6693.428718] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6700.979666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6701.224652] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6701.227447] LustreError: 185159:0:(llog.c:1657:llog_backup()) MGC192.168.206.113@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6701.239853] Lustre: 185159:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.206.113@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6701.386603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6707.155313] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6712.922790] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6715.874425] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6715.896466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6715.907202] Lustre: Skipped 1 previous similar message [ 6720.480530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6720.501623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6720.515685] Lustre: Skipped 1 previous similar message [ 6720.526482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6721.674862] LustreError: 185812:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6721.848740] Lustre: server umount lustre-MDT0000 complete [ 6726.113243] LustreError: 183518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6726.132751] LustreError: 183518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6726.372654] LustreError: 183499:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244101 with bad export cookie 1797777173114182164 [ 6726.398272] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6726.550534] LustreError: 186013:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6726.552975] LustreError: 186013:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6727.024889] Lustre: server umount lustre-MDT0001 complete [ 6741.303959] LustreError: 186214:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6741.319495] LustreError: 186214:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6741.610462] Lustre: server umount lustre-OST0000 complete [ 6751.382905] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 6762.421022] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 6772.501903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6779.056739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6785.418736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6791.185734] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6799.040132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6799.100461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6799.389385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6799.438697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6799.523623] Lustre: lustre-MDT0000: new disk, initializing [ 6799.611708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6799.632827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6803.980410] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6814.877741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6814.935633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6815.040362] Lustre: 188557:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6815.086114] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6815.088290] Lustre: Skipped 1 previous similar message [ 6815.090577] LustreError: 189324:0:(llog.c:1657:llog_backup()) MGC192.168.206.113@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6815.095492] LustreError: 189324:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 6815.098102] Lustre: 189324:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.206.113@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6815.118963] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6815.121774] Lustre: Skipped 1 previous similar message [ 6815.173404] Lustre: lustre-MDT0001: new disk, initializing [ 6815.258694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6815.302718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6815.308548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6815.376450] Lustre: 189352:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6815.381536] Lustre: 189352:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6815.391305] Lustre: 189352:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6815.400903] Lustre: 189352:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6815.415090] Lustre: 189352:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6815.418111] Lustre: 189352:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6818.708285] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6825.773882] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6828.953903] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6834.776844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6834.874756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6835.042904] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6835.044755] Lustre: Skipped 1 previous similar message [ 6835.053729] LustreError: 190280:0:(llog.c:1657:llog_backup()) MGC192.168.206.113@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6835.058264] LustreError: 190280:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 6835.069372] Lustre: 190280:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.206.113@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6835.079239] Lustre: 190280:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6835.094307] Lustre: lustre-OST0000: new disk, initializing [ 6835.100586] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6835.195673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6840.430397] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6844.989771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6844.995691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6845.074510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6846.804281] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6850.978944] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6851.252698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6855.068383] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6855.285985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6860.773576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6860.776977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6860.781132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6860.794018] Lustre: Skipped 3 previous similar messages [ 6863.259943] LustreError: 191499:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6863.263465] LustreError: 191499:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6863.421541] Lustre: server umount lustre-OST0000 complete [ 6871.008235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6871.030987] Lustre: Skipped 1 previous similar message [ 6871.034335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6876.135020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6876.137379] Lustre: Skipped 1 previous similar message [ 6881.247878] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6881.414209] LustreError: 191700:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6881.431194] LustreError: 191700:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6881.525307] Lustre: server umount lustre-MDT0000 complete [ 6885.488159] LustreError: 188549:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244260 with bad export cookie 1797777173114182885 [ 6885.491794] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6885.516987] LustreError: 188549:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6886.039893] Lustre: server umount lustre-MDT0001 complete [ 6895.329560] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 14:31:09 (1761244269) [ 6909.754607] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 6918.810410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6919.228650] LustreError: 193273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6919.288555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6922.589735] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6924.771363] LustreError: 193273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6929.175859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6932.156108] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6934.599830] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6940.109536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6945.610330] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6953.155590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6953.209179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6953.322042] Lustre: lustre-OST0001: new disk, initializing [ 6953.325496] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6953.427906] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6953.434897] Lustre: Skipped 2 previous similar messages [ 6958.097198] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 6961.737710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6961.744435] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6961.826345] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6966.463602] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6978.115089] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6983.358731] Lustre: 193268:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 6983.364618] Lustre: 193268:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 6983.375949] Lustre: 193268:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6983.384358] Lustre: 193268:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6983.393136] Lustre: 193268:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 6983.402657] Lustre: 193268:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6983.407919] Lustre: 193268:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 6983.412027] Lustre: 193268:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6983.417202] Lustre: 193268:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6983.427532] Lustre: 193268:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6983.433092] Lustre: 193268:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6983.437514] Lustre: 193268:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6984.856825] Lustre: DEBUG MARKER: ADD OST3 [ 6988.816550] loop: module loaded [ 6988.856885] loop0: detected capacity change from 0 to 204800000 [ 6989.060246] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6989.073796] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6989.082781] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6989.174409] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6994.007398] loop0: detected capacity change from 0 to 204800000 [ 6995.832644] Lustre: DEBUG MARKER: STOP OST3 [ 6999.955126] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6999.992642] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7000.290419] Lustre: lustre-OST0003: new disk, initializing [ 7000.296969] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7001.675001] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7001.849459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7001.863746] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7001.921689] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7002.050053] Lustre: Failing over lustre-OST0003 [ 7002.108137] LustreError: 197639:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 7002.110859] LustreError: 197639:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7002.232711] Lustre: server umount lustre-OST0003 complete [ 7003.680264] LustreError: 197596:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7007.035732] LustreError: 194732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7007.171811] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7007.218396] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7007.228430] Lustre: Skipped 2 previous similar messages [ 7007.244339] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7007.249539] LustreError: Skipped 1 previous similar message [ 7016.791217] Lustre: DEBUG MARKER: START OST3 again [ 7017.233866] LustreError: 194732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7017.249105] LustreError: 194732:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7023.141419] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7023.450788] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7023.457522] Lustre: Skipped 1 previous similar message [ 7023.484927] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7024.630360] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7025.080138] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7025.082034] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7028.329524] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7047.257180] Lustre: Failing over lustre-OST0003 [ 7047.295246] LustreError: 199873:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 7047.298195] LustreError: 199873:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7047.346673] Lustre: server umount lustre-OST0003 complete [ 7049.195275] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7049.201794] LustreError: 194732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7049.204628] Lustre: Skipped 1 previous similar message [ 7049.212685] LustreError: 194732:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 7064.557095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7064.559975] Lustre: Skipped 4 previous similar messages [ 7068.933618] Lustre: server umount lustre-MDT0000 complete [ 7071.851624] LustreError: 193253:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244446 with bad export cookie 1797777173114183809 [ 7071.856547] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7071.857713] LustreError: 193253:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7074.796595] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7074.816838] Lustre: Skipped 5 previous similar messages [ 7078.320527] Lustre: server umount lustre-MDT0001 complete [ 7081.258144] LustreError: 200550:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7081.270803] LustreError: 200550:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 7081.307924] Lustre: server umount lustre-OST0000 complete [ 7084.822892] Lustre: server umount lustre-OST0001 complete [ 7090.478569] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 7096.840700] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 7104.974751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7110.945571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7116.610596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7122.793392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7128.877488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7128.953199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7129.195178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7129.253315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7129.302555] Lustre: lustre-MDT0000: new disk, initializing [ 7129.405784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7133.074589] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7141.030488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7141.069894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7141.120953] Lustre: 202937:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7141.318362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7141.368556] Lustre: 203734:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 7141.376897] Lustre: 203734:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 2 previous similar messages [ 7141.379367] Lustre: 203734:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 7141.386682] Lustre: 203734:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 7141.391538] Lustre: 203734:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 7141.395432] Lustre: 203734:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 7141.403741] Lustre: 203734:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 7141.413260] Lustre: 203734:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 7141.418356] Lustre: 203734:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 7141.422670] Lustre: 203734:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 7141.428742] Lustre: 203734:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 7141.438060] Lustre: 203734:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 7143.576608] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7149.197778] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7152.897647] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7158.547355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7158.595856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7158.797633] Lustre: lustre-OST0000: new disk, initializing [ 7158.802659] Lustre: Skipped 1 previous similar message [ 7158.809191] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7158.818103] Lustre: Skipped 2 previous similar messages [ 7158.890545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7158.896748] Lustre: Skipped 2 previous similar messages [ 7160.364946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7160.378391] Lustre: Skipped 1 previous similar message [ 7160.383563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7160.500240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7164.282281] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7170.169637] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7173.183226] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7173.356471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7176.244307] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7176.418400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7180.771837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7180.775244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7180.778871] Lustre: Skipped 1 previous similar message [ 7180.783331] Lustre: Skipped 5 previous similar messages [ 7183.935415] LustreError: 205876:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7183.949460] LustreError: 205876:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 7184.084518] Lustre: server umount lustre-OST0000 complete [ 7193.847014] Lustre: server umount lustre-MDT0000 complete [ 7196.128407] LustreError: 202948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7196.137890] LustreError: 202948:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 7197.053800] LustreError: 202928:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244572 with bad export cookie 1797777173114186105 [ 7197.054630] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7197.064048] LustreError: 202928:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7197.420351] Lustre: server umount lustre-MDT0001 complete [ 7200.046179] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 14:36:14 (1761244574) [ 7211.701652] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 7219.291314] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 7228.359788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7233.189812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7237.831471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7242.325319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7247.118520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7247.163794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7247.293809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7247.319581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7247.383169] Lustre: lustre-MDT0000: new disk, initializing [ 7247.482441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7250.107467] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7257.334345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7257.384258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7257.445496] Lustre: 209122:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7257.629536] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7257.714821] Lustre: 209921:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 7257.733323] Lustre: 209921:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 7257.743759] Lustre: 209921:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 7257.756384] Lustre: 209921:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7257.761592] Lustre: 209921:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 7257.772977] Lustre: 209921:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7257.801211] Lustre: 209921:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 7257.806105] Lustre: 209921:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7257.809794] Lustre: 209921:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 7257.813351] Lustre: 209921:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7257.821480] Lustre: 209921:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 7257.825548] Lustre: 209921:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7260.312225] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7265.924789] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7268.236547] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7273.408614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7273.481404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7275.557059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7278.948598] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7287.455931] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7292.577734] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7292.782576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7297.728899] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7298.075533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7301.093633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7301.095305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7301.097313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7301.097321] Lustre: Skipped 2 previous similar messages [ 7301.116476] Lustre: Skipped 3 previous similar messages [ 7301.127148] LustreError: Skipped 1 previous similar message [ 7314.399230] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7314.463178] LustreError: 212063:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7314.469187] LustreError: 212063:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7314.638913] Lustre: server umount lustre-OST0000 complete [ 7319.011353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7324.415184] Lustre: server umount lustre-MDT0000 complete [ 7326.688922] LustreError: 209924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7327.863469] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7327.865594] LustreError: 209115:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244702 with bad export cookie 1797777173114187029 [ 7327.897145] LustreError: 209115:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7328.207494] Lustre: server umount lustre-MDT0001 complete [ 7335.447319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7339.279663] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7346.749312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7350.057899] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7354.133562] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7356.860717] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7361.809583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7366.521656] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7370.458449] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7391.203450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7391.209666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7391.215702] Lustre: Skipped 8 previous similar messages [ 7396.574589] Lustre: server umount lustre-OST0000 complete [ 7414.239211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7414.398201] Lustre: server umount lustre-MDT0000 complete [ 7417.554334] LustreError: 212924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244792 with bad export cookie 1797777173114187953 [ 7417.555930] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7417.572199] LustreError: 212924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7417.879831] Lustre: server umount lustre-MDT0001 complete [ 7505.086337] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 7507.061356] Key type lgssc unregistered [ 7507.283913] LNet: 216280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7507.287794] LNetError: 216280:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7507.300555] LNet: Removed LNI 192.168.206.113@tcp [ 7507.827324] Key type .llcrypt unregistered [ 7507.828937] Key type ._llcrypt unregistered [ 7526.527171] Key type ._llcrypt registered [ 7526.528494] Key type .llcrypt registered [ 7526.628561] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 7540.528935] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 7541.583027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7541.631513] alg: No test for adler32 (adler32-zlib) [ 7542.643595] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 7542.843895] LNet: Added LNI 192.168.206.113@tcp [8/256/0/180] [ 7544.527814] Key type lgssc registered [ 7545.862852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7554.572524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7559.948584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7565.325763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7570.967355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7576.792350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7576.853425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7576.868364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7578.113653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7578.139407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7578.192677] Lustre: lustre-MDT0000: new disk, initializing [ 7578.276858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7578.289202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7581.617524] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7592.206529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7592.322898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7592.474181] Lustre: 219608:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7592.521831] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7592.534060] Lustre: Skipped 1 previous similar message [ 7592.615572] Lustre: lustre-MDT0001: new disk, initializing [ 7592.738698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7592.795317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7592.809762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7592.907518] Lustre: 220406:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 7592.925791] Lustre: 220406:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 7592.929162] Lustre: 220406:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 7592.959471] Lustre: 220406:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 7592.979060] Lustre: 220406:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 7592.988485] Lustre: 220406:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 7596.198602] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7601.907911] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7604.372906] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7608.910507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7608.947061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7609.083194] Lustre: lustre-OST0000: new disk, initializing [ 7609.086656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7609.151169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7613.254251] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7618.585987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7618.590312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7618.669594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7619.106791] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7621.984850] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7622.147563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7624.728549] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7624.861560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7628.769309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7628.769940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7628.776155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7628.788531] LustreError: Skipped 1 previous similar message [ 7628.792428] Lustre: Skipped 1 previous similar message [ 7632.185145] LustreError: 222548:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7632.322612] Lustre: server umount lustre-OST0000 complete [ 7639.010614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7639.013641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7639.016493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7639.030525] Lustre: Skipped 2 previous similar messages [ 7644.128608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7644.131378] Lustre: Skipped 1 previous similar message [ 7648.230698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7649.249397] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7649.309353] LustreError: 222749:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7649.312229] LustreError: 222749:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7649.379955] Lustre: server umount lustre-MDT0000 complete [ 7651.964915] LustreError: 219600:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245027 with bad export cookie 1999594144639477968 [ 7651.968964] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7651.980343] LustreError: 219600:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7652.107780] LustreError: 222952:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7652.114046] LustreError: 222952:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7652.379907] Lustre: server umount lustre-MDT0001 complete [ 7665.831544] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 7673.547736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7673.802419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7676.687454] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7678.946734] LustreError: 224127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7678.959634] LustreError: 224127:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7683.042055] LustreError: 224127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7683.169703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7683.496658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7686.866508] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7689.459697] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7695.056846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7695.297162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7699.354110] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7706.665730] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7706.745917] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7706.930467] Lustre: lustre-OST0001: new disk, initializing [ 7706.935305] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7707.232666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7707.248049] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7707.312831] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7712.016887] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7720.714936] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7728.654359] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7743.455337] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 14:45:18 (1761245118) [ 7744.993481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7744.997550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7745.007694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7745.010854] Lustre: Skipped 1 previous similar message [ 7747.562493] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7747.572467] Lustre: Skipped 2 previous similar messages [ 7757.795440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7757.805159] Lustre: Skipped 11 previous similar messages [ 7759.327369] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7759.409498] LustreError: 227528:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7759.412567] LustreError: 227528:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7759.455897] Lustre: server umount lustre-MDT0000 complete [ 7762.163342] LustreError: 226393:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245137 with bad export cookie 1999594144639478892 [ 7762.164354] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7762.170474] LustreError: 226393:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7762.464778] Lustre: server umount lustre-MDT0001 complete [ 7775.702216] LustreError: 227932:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7775.704386] LustreError: 227932:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 7775.754958] Lustre: server umount lustre-OST0000 complete [ 7788.866219] LustreError: 228134:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7788.872267] LustreError: 228134:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7789.174061] Lustre: server umount lustre-OST0001 complete [ 7794.099340] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 7802.627281] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 7811.928575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7817.968752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7824.412452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7830.230730] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7835.788453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7835.833461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7836.017755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7836.048870] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7836.131608] Lustre: lustre-MDT0000: new disk, initializing [ 7836.246483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7836.253197] Lustre: Skipped 1 previous similar message [ 7836.266768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7839.732639] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7849.221016] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7849.293993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7849.387638] Lustre: 230321:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7849.450645] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7849.453933] Lustre: Skipped 1 previous similar message [ 7849.556760] Lustre: lustre-MDT0001: new disk, initializing [ 7849.712111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7849.722634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7852.917362] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7859.846363] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7862.949947] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7869.080575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7869.137296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7869.338247] Lustre: lustre-OST0000: new disk, initializing [ 7869.346816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7869.431523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7869.434209] Lustre: Skipped 1 previous similar message [ 7870.773171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7870.788051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7870.912651] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7874.710956] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7881.472323] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7885.509061] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7885.844532] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7889.686794] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7889.904638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7896.549920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7896.550315] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7896.597212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7906.271131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7906.302183] LustreError: 233260:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7906.318667] LustreError: 233260:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7906.520345] Lustre: server umount lustre-OST0000 complete [ 7911.393760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7911.402738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7911.407813] Lustre: Skipped 1 previous similar message [ 7924.191691] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7924.341526] Lustre: server umount lustre-MDT0000 complete [ 7926.249593] LustreError: 230328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7926.260219] LustreError: 230328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7927.906576] LustreError: 230314:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245303 with bad export cookie 1999594144639481034 [ 7927.914895] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7927.921525] LustreError: 230314:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7928.333421] Lustre: server umount lustre-MDT0001 complete [ 7936.394548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7936.885328] LustreError: 234142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7936.975078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7940.351750] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7942.113052] LustreError: 234142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7947.907460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7951.506738] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7956.206916] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7960.146685] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7967.097277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7972.119899] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 7977.244613] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8126.437741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8126.438523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8126.457086] Lustre: Skipped 1 previous similar message [ 8126.462055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8126.478580] Lustre: Skipped 10 previous similar messages [ 8130.630812] LustreError: 236430:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8130.656425] LustreError: 236430:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8130.987481] Lustre: server umount lustre-OST0000 complete [ 8141.809551] LustreError: 234142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8141.854654] LustreError: 234142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8142.065569] Lustre: server umount lustre-MDT0000 complete [ 8146.264261] LustreError: 234121:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245521 with bad export cookie 1999594144639481951 [ 8146.271138] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8146.289982] LustreError: 234121:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8146.810204] Lustre: server umount lustre-MDT0001 complete [ 8244.293233] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 8246.969755] Key type lgssc unregistered [ 8247.236315] LNet: 237400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8247.243215] LNetError: 237400:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8247.262787] LNet: Removed LNI 192.168.206.113@tcp [ 8248.035169] Key type .llcrypt unregistered [ 8248.037109] Key type ._llcrypt unregistered [ 8268.797692] Key type ._llcrypt registered [ 8268.799175] Key type .llcrypt registered [ 8268.896963] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 8281.744681] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 8282.682835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8282.713804] alg: No test for adler32 (adler32-zlib) [ 8283.735718] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 8283.947921] LNet: Added LNI 192.168.206.113@tcp [8/256/0/180] [ 8285.624946] Key type lgssc registered [ 8286.759964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8294.924404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8299.952413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8304.665036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8309.447702] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8323.701359] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 8332.902159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8332.952567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8332.970768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8334.203514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8334.235214] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8334.302522] Lustre: lustre-MDT0000: new disk, initializing [ 8334.414498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8334.441333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8338.391295] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8348.230689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8348.302821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8348.399828] Lustre: 241426:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8348.483730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8348.486507] Lustre: Skipped 1 previous similar message [ 8348.556176] Lustre: lustre-MDT0001: new disk, initializing [ 8348.644943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8348.680192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8348.683594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8348.762359] Lustre: 242225:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 8348.774859] Lustre: 242225:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 8348.786539] Lustre: 242225:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 8348.793318] Lustre: 242225:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 8348.802293] Lustre: 242225:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 8348.807101] Lustre: 242225:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 8351.990317] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8356.394883] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8362.204201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8362.276264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8362.464464] Lustre: lustre-OST0000: new disk, initializing [ 8362.467535] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8362.548608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8362.982601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8362.991732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8363.132275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8367.691868] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8376.996288] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8377.067826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8377.174663] Lustre: lustre-OST0001: new disk, initializing [ 8377.180642] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8377.274870] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8382.478244] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8384.622484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8384.629022] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8384.713949] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8392.926407] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8399.205606] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8415.201048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8415.213903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8415.222815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8415.721488] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8415.724407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8415.729488] Lustre: Skipped 1 previous similar message [ 8415.737478] Lustre: Skipped 2 previous similar messages [ 8419.758814] LustreError: 244953:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8419.806432] Lustre: server umount lustre-MDT0000 complete [ 8421.897227] LustreError: 242208:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245796 with bad export cookie 1854326697620085039 [ 8421.901686] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8421.907140] LustreError: 242208:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8422.001054] LustreError: 245154:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 8422.004572] LustreError: 245154:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8422.155125] Lustre: server umount lustre-MDT0001 complete [ 8434.204435] LustreError: 245354:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8434.213041] LustreError: 245354:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8434.280612] Lustre: server umount lustre-OST0000 complete [ 8447.766585] LustreError: 245556:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8447.772704] LustreError: 245556:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8447.934087] Lustre: server umount lustre-OST0001 complete [ 8459.136895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8467.293041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8468.267875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8474.717301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8481.330193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8482.269211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8498.409586] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 8507.104485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8507.116259] Lustre: lustre-MDT0000: reset Object Index mappings [ 8507.894278] LustreError: 248872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8507.948093] LustreError: 248872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8508.033463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8512.200947] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8513.508199] LustreError: 248871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8513.528233] LustreError: 248871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8518.635464] LustreError: 248871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8520.181467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8520.201911] Lustre: lustre-MDT0001: reset Object Index mappings [ 8520.536953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8523.918484] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8526.693314] Lustre: 249984:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8533.050227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8538.476726] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8545.487513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8545.663712] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8545.667640] Lustre: Skipped 1 previous similar message [ 8550.362631] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8553.910717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8553.918224] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8562.690617] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8565.695556] Lustre: 251833:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8574.436287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8574.438439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8574.441524] Lustre: Skipped 3 previous similar messages [ 8574.445492] Lustre: Skipped 1 previous similar message [ 8579.557991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8579.576068] Lustre: Skipped 5 previous similar messages [ 8580.279285] LustreError: 251937:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8580.283743] LustreError: 251937:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8580.351724] Lustre: server umount lustre-MDT0000 complete [ 8583.549531] LustreError: 248851:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245958 with bad export cookie 1854326697620091871 [ 8583.562303] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8583.570425] LustreError: 248851:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8583.880034] Lustre: server umount lustre-MDT0001 complete [ 8595.970940] LustreError: 252340:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 8595.973223] LustreError: 252340:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 8596.065080] Lustre: server umount lustre-OST0000 complete [ 8608.246705] Lustre: server umount lustre-OST0001 complete [ 8623.807538] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 14:59:58 (1761245998) [ 8637.174229] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 8648.554040] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 8657.663262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8663.916227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8669.231746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8675.022206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8680.784977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8680.861463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8681.043538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8681.073618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8681.149595] Lustre: lustre-MDT0000: new disk, initializing [ 8681.264954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8681.295517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8684.580402] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8694.443991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8694.493941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8694.571951] Lustre: 256067:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8694.629322] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8694.632362] Lustre: Skipped 1 previous similar message [ 8694.701648] Lustre: lustre-MDT0001: new disk, initializing [ 8694.832887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8694.843265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8694.958877] Lustre: 256877:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 8694.971114] Lustre: 256877:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 8694.982177] Lustre: 256877:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 8694.994626] Lustre: 256877:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8695.004890] Lustre: 256877:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 8695.016190] Lustre: 256877:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8695.027173] Lustre: 256877:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 8695.044539] Lustre: 256877:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8695.067650] Lustre: 256877:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 8695.071180] Lustre: 256877:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8695.075731] Lustre: 256877:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 8695.079351] Lustre: 256877:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8698.817282] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8705.025428] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8708.155271] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8713.839162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8713.921809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8714.081685] Lustre: lustre-OST0000: new disk, initializing [ 8714.087842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8715.384908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8715.398106] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8715.476367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8718.682548] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8724.466352] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8728.197993] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8728.470178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8732.090797] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8732.268433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8736.230866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8736.233717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8736.259031] Lustre: Skipped 1 previous similar message [ 8736.277139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8736.279842] Lustre: Skipped 1 previous similar message [ 8739.979115] LustreError: 259018:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8739.982097] LustreError: 259018:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 8740.189213] Lustre: server umount lustre-OST0000 complete [ 8745.953758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8745.972538] LustreError: Skipped 1 previous similar message [ 8745.975496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8745.984391] Lustre: Skipped 1 previous similar message [ 8745.988834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8749.575813] Lustre: server umount lustre-MDT0000 complete [ 8751.587183] LustreError: 256078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8751.616023] LustreError: 256078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8752.745677] LustreError: 256859:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246127 with bad export cookie 1854326697620093964 [ 8752.749375] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8752.756117] LustreError: 256859:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8753.130208] Lustre: server umount lustre-MDT0001 complete [ 8768.028472] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 8776.809351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8777.071800] LustreError: 260596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8777.133366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8777.136529] Lustre: Skipped 2 previous similar messages [ 8780.621954] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8787.441156] LustreError: 260596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8787.452724] LustreError: 260596:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8788.733371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8792.443029] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8795.285838] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8801.105845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8806.476258] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8814.137821] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8814.250860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8814.450235] Lustre: lustre-OST0001: new disk, initializing [ 8814.464264] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8815.633594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8815.646036] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8815.767523] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8819.092492] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8827.372976] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8830.386563] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8840.219414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8840.274317] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8845.281102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8845.284580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8845.291187] Lustre: Skipped 1 previous similar message [ 8845.295588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8845.298363] Lustre: Skipped 2 previous similar messages [ 8847.363510] LustreError: 264104:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8847.384030] LustreError: 264104:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8847.508853] Lustre: server umount lustre-MDT0000 complete [ 8850.421490] LustreError: 260577:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246225 with bad export cookie 1854326697620094888 [ 8850.423651] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8850.439721] LustreError: 260577:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8850.687687] Lustre: server umount lustre-MDT0001 complete [ 8863.727848] Lustre: server umount lustre-OST0000 complete [ 8876.836351] Lustre: server umount lustre-OST0001 complete [ 8891.787660] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 8899.894040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8900.167825] LustreError: 265884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8904.259643] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8911.704778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8912.083367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8912.092200] Lustre: Skipped 4 previous similar messages [ 8915.695261] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8917.945022] Lustre: 266994:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8924.349580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8925.760826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 8929.435454] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8935.908661] LustreError: 267352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8935.915344] LustreError: 267352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8936.767754] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8938.868826] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 8941.773333] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 8947.938242] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8950.880595] Lustre: 268852:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8959.921415] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8959.965417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8969.528430] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 15:05:43 (1761246343) [ 8983.883313] LustreError: 269560:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8983.888521] LustreError: 269560:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 8984.051677] Lustre: server umount lustre-MDT0000 complete [ 8987.925284] LustreError: 265864:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246363 with bad export cookie 1854326697620101181 [ 8987.942501] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8987.944872] LustreError: 265864:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8988.642290] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8988.654143] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8988.663733] Lustre: Skipped 6 previous similar messages [ 8988.671352] Lustre: Skipped 4 previous similar messages [ 8994.508587] Lustre: server umount lustre-MDT0001 complete [ 9013.218158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9013.322108] Lustre: server umount lustre-OST0000 complete [ 9016.968337] Lustre: server umount lustre-OST0001 complete [ 9022.560577] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 9031.760972] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 9039.629272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9045.082915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9050.626040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9056.441525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9072.601810] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 9081.536424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9081.606363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9081.820113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9081.852754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9081.929738] Lustre: lustre-MDT0000: new disk, initializing [ 9082.021432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9085.816578] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 9095.358403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9095.418800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9095.504406] Lustre: 273050:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9095.799730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9095.918860] Lustre: 273853:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 9095.936838] Lustre: 273853:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 9095.945106] Lustre: 273853:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 9095.947490] Lustre: 273853:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 9095.950379] Lustre: 273853:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 9095.953275] Lustre: 273853:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 9095.956418] Lustre: 273853:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 9095.959038] Lustre: 273853:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 9095.963792] Lustre: 273853:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 9095.974091] Lustre: 273853:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 9096.004786] Lustre: 273853:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 9096.022366] Lustre: 273853:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 9099.410343] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 9110.396748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9110.460416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9112.733064] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9116.045265] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 9125.920625] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9125.993831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9127.947161] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9127.959867] Lustre: Skipped 1 previous similar message [ 9128.133559] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9132.576587] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 9142.532744] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9146.271474] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9146.274212] Lustre: Skipped 1 previous similar message [ 9165.122626] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 15:08:59 (1761246539) [ 9168.866436] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9168.872907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9168.892841] Lustre: Skipped 1 previous similar message [ 9168.924858] Lustre: Skipped 3 previous similar messages [ 9169.652971] LustreError: 276669:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 9169.657309] LustreError: 276669:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 9169.842493] Lustre: server umount lustre-MDT0000 complete [ 9173.679145] LustreError: 273041:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246548 with bad export cookie 1854326697620109056 [ 9173.687672] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9173.987097] LustreError: 273056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9174.003452] LustreError: 273056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 9174.346498] Lustre: server umount lustre-MDT0001 complete [ 9180.509504] Lustre: server umount lustre-OST0000 complete [ 9184.582269] Lustre: server umount lustre-OST0001 complete [ 9203.083217] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 9214.029313] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 9221.330474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9226.727331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9232.574916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9239.065587] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9246.458328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9246.562708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9246.776175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9246.806265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9246.809333] Lustre: Skipped 4 previous similar messages [ 9246.858184] Lustre: lustre-MDT0000: new disk, initializing [ 9246.860160] Lustre: Skipped 3 previous similar messages [ 9246.939578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9246.942510] Lustre: Skipped 6 previous similar messages [ 9246.960303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9246.964978] Lustre: Skipped 3 previous similar messages [ 9251.069891] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 9261.858808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9261.931882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9262.031245] Lustre: 280115:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9262.427552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9262.548719] Lustre: 280913:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 9262.557226] Lustre: 280913:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 9262.565790] Lustre: 280913:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 9262.580708] Lustre: 280913:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 9262.598773] Lustre: 280913:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 9262.611812] Lustre: 280913:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 9262.629589] Lustre: 280913:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 9262.640627] Lustre: 280913:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 9262.645089] Lustre: 280913:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 9262.651327] Lustre: 280913:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 9262.657475] Lustre: 280913:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 9262.664271] Lustre: 280913:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 9266.145381] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 9273.604352] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9277.634816] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9284.702977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9284.778713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9286.570805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9291.283224] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 9298.230676] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9300.348887] Lustre: Mounted lustre-client [ 9301.259793] LustreError: 282624:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9c0d4e9f8800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 9301.317240] Lustre: Unmounted lustre-client [ 9310.693717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9310.703128] Lustre: Skipped 4 previous similar messages [ 9310.709636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9310.715285] Lustre: Skipped 1 previous similar message [ 9316.806810] Lustre: server umount lustre-OST0000 complete [ 9327.269920] Lustre: server umount lustre-MDT0000 complete [ 9331.173389] LustreError: 281273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9331.208338] LustreError: 281273:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9331.397334] LustreError: 280108:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246706 with bad export cookie 1854326697620110624 [ 9331.398517] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9331.410521] LustreError: 280108:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9331.894373] Lustre: server umount lustre-MDT0001 complete [ 9342.432406] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 9345.225691] Key type lgssc unregistered [ 9361.951974] LNet: 283903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9361.969671] LNetError: 283903:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9430.630208] LNet: Removed LNI 192.168.206.113@tcp [ 9431.306298] Key type .llcrypt unregistered [ 9431.310456] Key type ._llcrypt unregistered [ 9449.338390] Key type ._llcrypt registered [ 9449.340156] Key type .llcrypt registered [ 9449.481531] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [ 9465.866037] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [ 9466.519981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9466.592921] alg: No test for adler32 (adler32-zlib) [ 9467.700945] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 9467.905094] LNet: Added LNI 192.168.206.113@tcp [8/256/0/180] [ 9469.543239] Key type lgssc registered [ 9470.619822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9479.453803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9486.050491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9492.126834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9498.899932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9500.867187] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 15:14:34 (1761246874) [ 9507.385377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9507.449745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9507.472557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9508.710291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9508.749652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9508.818997] Lustre: lustre-MDT0000: new disk, initializing [ 9508.888266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9508.912673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9512.361521] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 9522.259411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9522.365282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9522.473670] Lustre: 287098:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9522.557215] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9522.569200] Lustre: Skipped 1 previous similar message [ 9522.676103] Lustre: lustre-MDT0001: new disk, initializing [ 9522.790150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9522.834374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9522.846996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9522.909665] Lustre: 287900:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 9522.922939] Lustre: 287900:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 9522.933610] Lustre: 287900:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 9522.942087] Lustre: 287900:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 9522.950107] Lustre: 287900:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 9522.956700] Lustre: 287900:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 9526.913484] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 9534.514155] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9538.955555] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9546.252826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9546.319434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9546.607762] Lustre: lustre-OST0000: new disk, initializing [ 9546.626499] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9546.798638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9551.689237] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_default_debug -1 all [ 9554.978277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9554.983397] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9555.136764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9558.596915] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9568.764139] LustreError: 289544:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9568.904561] Lustre: server umount lustre-OST0000 complete [ 9573.860576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9573.873225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9573.903566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9575.910145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9575.914365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9581.024194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9581.031555] Lustre: Skipped 2 previous similar messages [ 9582.055379] LustreError: 289744:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9582.058082] LustreError: 289744:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9582.120932] Lustre: server umount lustre-MDT0000 complete [ 9584.923263] LustreError: 287090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246960 with bad export cookie 3789002895798003658 [ 9584.927324] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9584.934915] LustreError: 287090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9595.370400] LustreError: 289946:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9595.373831] LustreError: 289946:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9595.587885] Lustre: server umount lustre-MDT0001 complete [ 9605.620686] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 9607.818958] Key type lgssc unregistered [ 9608.050251] LNet: 290479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9608.056463] LNetError: 290479:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9608.073176] LNet: Removed LNI 192.168.206.113@tcp [ 9608.794647] Key type .llcrypt unregistered [ 9608.797412] Key type ._llcrypt unregistered [ 9689.088129] Key type ._llcrypt registered [ 9689.092257] Key type .llcrypt registered [ 9689.210397] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 9689.431201] Key type .llcrypt unregistered [ 9689.433118] Key type ._llcrypt unregistered [ 9740.497709] Key type ._llcrypt registered [ 9740.499260] Key type .llcrypt registered [ 9740.591226] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 9740.796041] Key type .llcrypt unregistered [ 9740.797682] Key type ._llcrypt unregistered [ 9767.873791] Key type ._llcrypt registered [ 9767.875725] Key type .llcrypt registered [ 9767.974380] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 9768.260330] Key type .llcrypt unregistered [ 9768.267468] Key type ._llcrypt unregistered [ 9835.456642] Key type ._llcrypt registered [ 9835.457991] Key type .llcrypt registered [ 9835.582251] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 9835.870688] Key type .llcrypt unregistered [ 9835.873993] Key type ._llcrypt unregistered [ 9863.335403] Key type ._llcrypt registered [ 9863.337294] Key type .llcrypt registered [ 9863.433299] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 9863.671400] Key type .llcrypt unregistered [ 9863.673253] Key type ._llcrypt unregistered [ 9888.936826] Key type ._llcrypt registered [ 9888.938767] Key type .llcrypt registered [ 9889.040970] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 9889.357896] Key type .llcrypt unregistered [ 9889.361207] Key type ._llcrypt unregistered [ 9955.487710] Key type ._llcrypt registered [ 9955.488969] Key type .llcrypt registered [ 9955.559574] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 9955.782353] Key type .llcrypt unregistered [ 9955.784246] Key type ._llcrypt unregistered [ 9981.001174] Key type ._llcrypt registered [ 9981.002901] Key type .llcrypt registered [ 9981.092424] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [ 9981.266356] Key type .llcrypt unregistered [ 9981.268050] Key type ._llcrypt unregistered [10004.470524] Key type ._llcrypt registered [10004.472368] Key type .llcrypt registered [10004.537940] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [10004.714731] Key type .llcrypt unregistered [10004.716781] Key type ._llcrypt unregistered [10053.303526] Key type ._llcrypt registered [10053.308637] Key type .llcrypt registered [10053.418517] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing set_hostid [10065.954657] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing load_modules_local [10066.564703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10066.576039] alg: No test for adler32 (adler32-zlib) [10067.716142] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [10067.960046] LNet: Added LNI 192.168.206.113@tcp [8/256/0/180] [10069.631176] Key type lgssc registered [10070.598421] Lustre: Echo OBD driver; http://www.lustre.org/ [10079.274630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10085.829253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10092.366320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10099.288372] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10101.174293] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9774 sec ========== 15:24:35 (1761247475) [10102.801652] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:24:36 (1761247476) === [10106.015908] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:24:40 (1761247480) === [10145.138125] Lustre: DEBUG MARKER: oleg613-server.virtnet: executing unload_modules_local [10147.929637] Key type lgssc unregistered [10148.178950] LNet: 309440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10148.190094] LNetError: 309440:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10148.204736] LNet: Removed LNI 192.168.206.113@tcp [10148.910460] Key type .llcrypt unregistered [10148.912264] Key type ._llcrypt unregistered