[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 709942215 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002351] x2apic enabled [ 0.003000] Switched APIC routing to physical x2apic. [ 0.003016] kvm-guest: setup PV IPIs [ 0.008485] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010009] pid_max: default: 32768 minimum: 301 [ 0.011144] LSM: Security Framework initializing [ 0.012048] Yama: becoming mindful. [ 0.013029] SELinux: Initializing. [ 0.014072] *** VALIDATE selinux *** [ 0.022747] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028340] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030096] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032112] *** VALIDATE tmpfs *** [ 0.034103] *** VALIDATE proc *** [ 0.035508] *** VALIDATE cgroup *** [ 0.036010] *** VALIDATE cgroup2 *** [ 0.037000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040029] Spectre V2 : User space: Vulnerable [ 0.041012] Speculative Store Bypass: Vulnerable [ 0.045202] debug: unmapping init [mem 0xffffffffae459000-0xffffffffae460fff] [ 0.048152] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049564] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050017] ... version: 2 [ 0.050810] ... bit width: 48 [ 0.051014] ... generic registers: 4 [ 0.052011] ... value mask: 0000ffffffffffff [ 0.053011] ... max period: 00007fffffffffff [ 0.054010] ... fixed-purpose events: 3 [ 0.055008] ... event mask: 000000070000000f [ 0.056262] rcu: Hierarchical SRCU implementation. [ 0.058268] smp: Bringing up secondary CPUs ... [ 0.059508] x86: Booting SMP configuration: [ 0.060027] .... node #0, CPUs: #1 #2 #3 [ 0.087181] smp: Brought up 1 node, 4 CPUs [ 0.089022] smpboot: Max logical packages: 1 [ 0.090009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.140025] node 0 deferred pages initialised in 40ms [ 0.150090] devtmpfs: initialized [ 0.151266] x86/mm: Memory block size: 128MB [ 0.155736] gcov: version magic: 0x41383552 [ 0.160352] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.166359] pinctrl core: initialized pinctrl subsystem [ 0.171346] [ 0.171779] ************************************************************* [ 0.172000] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.184020] ** ** [ 0.199014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.206021] ** ** [ 0.208016] ** This means that this kernel is built to expose internal ** [ 0.213044] ** IOMMU data structures, which may compromise security on ** [ 0.215011] ** your system. ** [ 0.229015] ** ** [ 0.231068] ** If you see this message and you are not debugging the ** [ 0.241013] ** kernel, report this immediately to your vendor! ** [ 0.254013] ** ** [ 0.256011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.265014] ************************************************************* [ 0.273216] NET: Registered protocol family 16 [ 0.276495] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.278054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.294068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.303072] cpuidle: using governor menu [ 0.308737] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.315079] PCI: Using configuration type 1 for base access [ 0.318126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.335473] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.336024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.337456] cryptd: max_cpu_qlen set to 1000 [ 0.340583] ACPI: Added _OSI(Module Device) [ 0.345024] ACPI: Added _OSI(Processor Device) [ 0.346022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.349024] ACPI: Added _OSI(Processor Aggregator Device) [ 0.353791] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.362489] ACPI: Interpreter enabled [ 0.365151] ACPI: PM: (supports S0 S3 S4 S5) [ 0.367084] ACPI: Using IOAPIC for interrupt routing [ 0.370179] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.373444] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.384153] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.385000] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.386017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.390098] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.393184] acpiphp: Slot [2] registered [ 0.395147] acpiphp: Slot [3] registered [ 0.396064] acpiphp: Slot [4] registered [ 0.398079] acpiphp: Slot [5] registered [ 0.399143] acpiphp: Slot [6] registered [ 0.400112] acpiphp: Slot [7] registered [ 0.402135] acpiphp: Slot [8] registered [ 0.404113] acpiphp: Slot [9] registered [ 0.405114] acpiphp: Slot [10] registered [ 0.407129] acpiphp: Slot [11] registered [ 0.408129] acpiphp: Slot [12] registered [ 0.410092] acpiphp: Slot [13] registered [ 0.411100] acpiphp: Slot [14] registered [ 0.413102] acpiphp: Slot [15] registered [ 0.414076] acpiphp: Slot [16] registered [ 0.416103] acpiphp: Slot [17] registered [ 0.418118] acpiphp: Slot [18] registered [ 0.422183] acpiphp: Slot [19] registered [ 0.425129] acpiphp: Slot [20] registered [ 0.427233] acpiphp: Slot [21] registered [ 0.429087] acpiphp: Slot [22] registered [ 0.430076] acpiphp: Slot [23] registered [ 0.434171] acpiphp: Slot [24] registered [ 0.435000] acpiphp: Slot [25] registered [ 0.435000] acpiphp: Slot [26] registered [ 0.437134] acpiphp: Slot [27] registered [ 0.442239] acpiphp: Slot [28] registered [ 0.444161] acpiphp: Slot [29] registered [ 0.445096] acpiphp: Slot [30] registered [ 0.451276] acpiphp: Slot [31] registered [ 0.453070] PCI host bridge to bus 0000:00 [ 0.454021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.455021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.460043] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.463024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.466032] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.469031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.471193] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.476590] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.480970] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.491834] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.498037] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.501018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.504023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.510022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.512495] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.515823] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.520048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.523870] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.528014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.536019] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.543016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.547240] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.556020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.563019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.585000] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.598765] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.604028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.607022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.630019] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.640707] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.648020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.658019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.678019] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.688000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.692017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.698020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.709016] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.722000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.725015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.729018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.740022] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.746861] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.752000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.755019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.765018] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.773571] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.775324] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.777288] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.778362] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.780196] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.784488] iommu: Default domain type: Passthrough [ 0.790488] SCSI subsystem initialized [ 0.791145] ACPI: bus type USB registered [ 0.793114] usbcore: registered new interface driver usbfs [ 0.796308] usbcore: registered new interface driver hub [ 0.798115] usbcore: registered new device driver usb [ 0.801636] pps_core: LinuxPPS API ver. 1 registered [ 0.804014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.808272] PTP clock support registered [ 0.811200] EDAC MC: Ver: 3.0.0 [ 0.815337] PCI: Using ACPI for IRQ routing [ 0.821568] NetLabel: Initializing [ 0.826016] NetLabel: domain hash size = 128 [ 0.830016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.831000] NetLabel: unlabeled traffic allowed by default [ 0.834053] vgaarb: loaded [ 0.835262] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.837009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.856815] clocksource: Switched to clocksource kvm-clock [ 0.987759] VFS: Disk quotas dquot_6.6.0 [ 0.989183] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.991610] *** VALIDATE ramfs *** [ 0.992652] *** VALIDATE hugetlbfs *** [ 0.994491] pnp: PnP ACPI init [ 0.996538] pnp: PnP ACPI: found 6 devices [ 1.013447] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.016280] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.018099] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.020724] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.022994] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.025393] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.028463] NET: Registered protocol family 2 [ 1.031749] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.037716] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.040677] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.045089] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.048592] TCP: Hash tables configured (established 65536 bind 65536) [ 1.051099] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.054578] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.057061] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.061242] NET: Registered protocol family 1 [ 1.063541] RPC: Registered named UNIX socket transport module. [ 1.065195] RPC: Registered udp transport module. [ 1.066656] RPC: Registered tcp transport module. [ 1.068132] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.070132] NET: Registered protocol family 44 [ 1.071490] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.073193] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.074973] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.077496] PCI: CLS 0 bytes, default 64 [ 1.079669] Unpacking initramfs... [ 2.864214] debug: unmapping init [mem 0xffff89c9fcc54000-0xffff89c9fffbffff] [ 2.869565] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.872495] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.876388] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.647198] Initialise system trusted keyrings [ 3.648728] Key type blacklist registered [ 3.650267] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.668387] zbud: loaded [ 3.676628] *** VALIDATE nfs *** [ 3.678098] *** VALIDATE nfs4 *** [ 3.679759] pstore: using deflate compression [ 3.683035] Platform Keyring initialized [ 3.893059] NET: Registered protocol family 38 [ 3.901977] Key type asymmetric registered [ 3.903437] Asymmetric key parser 'x509' registered [ 3.905515] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.909407] io scheduler mq-deadline registered [ 3.913467] io scheduler kyber registered [ 3.915373] io scheduler bfq registered [ 3.918686] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.923397] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.930914] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.934761] ACPI: Power Button [PWRF] [ 4.044296] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.156476] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.365316] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.470983] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.762951] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.801677] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.847070] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.856332] Non-volatile memory driver v1.3 [ 4.857687] Linux agpgart interface v0.103 [ 4.887708] virtio_blk virtio1: [vda] 133928 512-byte logical blocks (68.6 MB/65.4 MiB) [ 4.890641] vda: detected capacity change from 0 to 68571136 [ 4.907458] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.910394] vdb: detected capacity change from 0 to 1073741824 [ 4.927786] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.930853] vdc: detected capacity change from 0 to 2621440000 [ 4.947475] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.951176] vdd: detected capacity change from 0 to 2621440000 [ 4.970406] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.975315] vde: detected capacity change from 0 to 4294967296 [ 4.993268] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.996373] vdf: detected capacity change from 0 to 4294967296 [ 5.005944] libphy: Fixed MDIO Bus: probed [ 5.012307] usbcore: registered new interface driver usbserial_generic [ 5.017962] usbserial: USB Serial support registered for generic [ 5.019963] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.028835] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.030096] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.031803] mousedev: PS/2 mouse device common for all mice [ 5.033709] rtc_cmos 00:05: RTC can wake from S4 [ 5.039228] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.043317] rtc_cmos 00:05: registered as rtc0 [ 5.050405] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.054878] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.055299] intel_pstate: CPU model not supported [ 5.062984] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.068049] hid: raw HID events driver (C) Jiri Kosina [ 5.072420] usbcore: registered new interface driver usbhid [ 5.074641] usbhid: USB HID core driver [ 5.079543] drop_monitor: Initializing network drop monitor service [ 5.081849] Initializing XFRM netlink socket [ 5.083620] NET: Registered protocol family 10 [ 5.088533] Segment Routing with IPv6 [ 5.090864] NET: Registered protocol family 17 [ 5.095532] mpls_gso: MPLS GSO support [ 5.100895] RAS: Correctable Errors collector initialized. [ 5.107042] AVX version of gcm_enc/dec engaged. [ 5.108805] AES CTR mode by8 optimization enabled [ 5.243709] sched_clock: Marking stable (5243448491, 0)->(6743093781, -1499645290) [ 5.248930] registered taskstats version 1 [ 5.250764] Loading compiled-in X.509 certificates [ 5.259145] zswap: loaded using pool lzo/zbud [ 5.307899] Key type big_key registered [ 5.334742] Key type encrypted registered [ 5.336562] ima: No TPM chip found, activating TPM-bypass! [ 5.340814] ima: Allocated hash algorithm: sha1 [ 5.342482] ima: No architecture policies found [ 5.344316] evm: Initialising EVM extended attributes: [ 5.349705] evm: security.selinux [ 5.351254] evm: security.ima [ 5.352578] evm: security.capability [ 5.354132] evm: HMAC attrs: 0x1 [ 5.358484] rtc_cmos 00:05: setting system clock to 2025-10-24 05:38:29 UTC (1761284309) [ 5.372230] debug: unmapping init [mem 0xffffffffaf403000-0xffffffffaf5fffff] [ 5.376130] debug: unmapping init [mem 0xffffffffae182000-0xffffffffae458fff] [ 5.394057] Write protecting the kernel read-only data: 28672k [ 5.401202] debug: unmapping init [mem 0xffffffffac803000-0xffffffffac9fffff] [ 5.403995] debug: unmapping init [mem 0xffffffffad114000-0xffffffffad1fffff] [ 5.467876] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.476370] systemd[1]: Detected virtualization kvm. [ 5.478331] systemd[1]: Detected architecture x86-64. [ 5.479932] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.510438] systemd[1]: No hostname configured. [ 5.512315] systemd[1]: Set hostname to . [ 5.513992] random: systemd: uninitialized urandom read (16 bytes read) [ 5.516384] systemd[1]: Initializing machine ID from random generator. [ 5.604478] random: ln: uninitialized urandom read (6 bytes read) [ 5.785772] random: systemd: uninitialized urandom read (16 bytes read) [ 5.788030] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.795639] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.804868] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.063666] device-mapper: uevent: version 1.0.3 [ 7.066506] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 7.901621] random: fast init done [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.092248] virtio_net virtio0 ens2: renamed from eth0 [ 8.347824] scsi host0: ata_piix [ 8.370531] scsi host1: ata_piix [ 8.372337] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.377405] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.867208] random: crng init done [ 12.868553] random: 7 urandom warning(s) missed due to ratelimiting [ 13.542590] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.536932] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.532991] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.053972] SELinux: Disabled at runtime. [ 17.144318] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.157604] systemd[1]: Detected virtualization kvm. [ 17.159568] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.220661] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.229947] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.248536] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.266925] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.277764] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.301484] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.326532] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 18.443074] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 19.227845] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.865771] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.134117] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.308950] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.340648] EDAC sbridge: Ver: 1.1.2 [ 23.083820] Key type dns_resolver registered [ 23.464886] NFS: Registering the id_resolver key type [ 23.472815] Key type id_resolver registered [ 23.475428] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg302-server login: [ 52.091286] libcfs: loading out-of-tree module taints kernel. [ 52.105734] Key type ._llcrypt registered [ 52.107175] Key type .llcrypt registered [ 52.156210] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 59.736695] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 60.367907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 60.374893] alg: No test for adler32 (adler32-zlib) [ 61.367241] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 61.686061] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 63.311217] Key type lgssc registered [ 63.906673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 70.692440] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 71.077028] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 74.113902] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 76.949722] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.787760] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 85.906041] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 90.985684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 91.011971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 91.020242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 92.115944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 92.130984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 92.173600] Lustre: lustre-MDT0000: new disk, initializing [ 92.210578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 92.221353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 93.698525] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 99.449732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.487656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 99.521801] Lustre: 6480:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 99.536214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 99.538735] Lustre: Skipped 1 previous similar message [ 99.582181] Lustre: lustre-MDT0001: new disk, initializing [ 99.612480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 99.627283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 99.633087] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 99.652527] LustreError: 7432:0:(llog_cat.c:511:llog_cat_current_log()) lustre-MDT0001-osd: next log does not exist, catalog [0x240000400:0x1:0x0] rc=-5 [ 99.658704] LustreError: 7432:0:(update_trans.c:965:top_trans_stop()) lustre-MDT0001-osd: write updates failed: rc = -5 [ 99.663713] LustreError: 7432:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 101.105673] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 103.603575] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 107.244825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.273169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 107.368749] Lustre: lustre-OST0000: new disk, initializing [ 107.371663] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 107.392640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 109.541294] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 115.184219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 115.186794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 115.213188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 115.312951] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 115.338681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 115.376797] Lustre: lustre-OST0001: new disk, initializing [ 115.378960] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 115.396541] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 117.381932] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 120.812736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 120.816474] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 120.828919] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 123.350853] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 127.052807] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 134.181751] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing check_logdir /tmp/testlogs/ [ 135.798427] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing yml_node [ 137.651918] Lustre: DEBUG MARKER: Client: 2.16.59.49 [ 138.705254] Lustre: DEBUG MARKER: MDS: 2.16.59.49 [ 139.804414] Lustre: DEBUG MARKER: OSS: 2.16.59.49 [ 140.536392] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 01:40:43 EDT 2025 [ 148.181466] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 148.926120] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 161.247996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 161.251571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 161.257079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 161.760403] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 161.761619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 161.771030] Lustre: Skipped 2 previous similar messages [ 161.773775] Lustre: Skipped 1 previous similar message [ 166.879767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 166.885369] Lustre: Skipped 4 previous similar messages [ 167.296863] LustreError: 12183:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 167.328966] Lustre: server umount lustre-MDT0000 complete [ 168.886246] LustreError: 7416:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284473 with bad export cookie 7401652262629013526 [ 168.888346] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 168.892386] LustreError: 7416:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 172.000880] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 172.001150] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 172.006221] Lustre: Skipped 1 previous similar message [ 172.010874] Lustre: Skipped 1 previous similar message [ 179.167506] LustreError: 6487:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 179.167534] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 179.174032] LustreError: 6487:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 6 previous similar messages [ 179.176389] Lustre: Skipped 3 previous similar messages [ 183.263197] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 183.303581] LustreError: 12385:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 183.306806] LustreError: 12385:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 183.402790] Lustre: server umount lustre-MDT0001 complete [ 184.981104] LustreError: 12587:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 184.984089] LustreError: 12587:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 185.009657] Lustre: server umount lustre-OST0000 complete [ 186.624909] Lustre: server umount lustre-OST0001 complete [ 189.054953] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 192.650203] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 196.616301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 199.098666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 201.429323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 203.907931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 206.581115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 206.610108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 206.718503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 206.734341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 206.773785] Lustre: lustre-MDT0000: new disk, initializing [ 206.812195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 206.821103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 208.527645] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 215.213884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 215.304471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 215.415113] Lustre: 14965:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 215.502988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 215.511802] Lustre: Skipped 1 previous similar message [ 215.598568] Lustre: lustre-MDT0001: new disk, initializing [ 215.715207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 215.769536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 215.785821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 215.891512] Lustre: 15763:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 215.908673] Lustre: 15763:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 215.924953] Lustre: 15763:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 215.936467] Lustre: 15763:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 215.950211] Lustre: 15763:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 215.957871] Lustre: 15763:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 220.017471] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 231.658385] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 238.163033] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 250.103764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 250.227392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 250.532951] Lustre: lustre-OST0000: new disk, initializing [ 250.539984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 250.656440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 252.027558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 252.051628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 252.208563] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 256.892138] hrtimer: interrupt took 6275820 ns [ 257.609752] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 265.361474] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 269.726473] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 269.953108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 274.142834] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 274.369438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 277.985619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 277.989600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 278.021439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 282.245977] LustreError: 17910:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 282.260073] LustreError: 17910:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 282.448377] Lustre: server umount lustre-OST0000 complete [ 287.712614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 287.718551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.731624] Lustre: Skipped 1 previous similar message [ 291.606360] LustreError: 18111:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 291.609799] LustreError: 18111:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 291.668642] Lustre: server umount lustre-MDT0000 complete [ 293.351590] LustreError: 14977:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 293.368584] LustreError: 14977:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 294.696857] LustreError: 14958:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284598 with bad export cookie 7401652262629015122 [ 294.698584] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 294.705517] LustreError: 14958:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 295.029797] Lustre: server umount lustre-MDT0001 complete [ 298.418422] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:43:20 (1761284600) [ 300.278316] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 301.731911] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:43:23 (1761284603) [ 307.085388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 307.377133] LustreError: 18983:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.454843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 310.514142] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 312.805265] LustreError: 18983:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.898924] LustreError: 18983:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.963921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 320.195637] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 324.622658] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 327.375433] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 332.049079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 336.397762] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 340.936918] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 353.257158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 353.260831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 353.304816] Lustre: Skipped 1 previous similar message [ 353.315604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 353.318639] Lustre: Skipped 4 previous similar messages [ 356.872134] LustreError: 21232:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 356.878568] LustreError: 21232:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 357.050669] Lustre: server umount lustre-OST0000 complete [ 363.493116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 366.795290] Lustre: server umount lustre-MDT0000 complete [ 368.609077] LustreError: 20240:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 368.631502] LustreError: 20240:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 370.191720] LustreError: 18963:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284674 with bad export cookie 7401652262629016046 [ 370.194495] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 370.211739] LustreError: 18963:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 370.499856] Lustre: server umount lustre-MDT0001 complete [ 379.591856] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 381.891146] Key type lgssc unregistered [ 382.148459] LNet: 22167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.153849] LNetError: 22167:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 382.175269] LNet: Removed LNI 192.168.203.102@tcp [ 383.001137] Key type .llcrypt unregistered [ 383.002806] Key type ._llcrypt unregistered [ 400.394900] Key type ._llcrypt registered [ 400.399038] Key type .llcrypt registered [ 400.495110] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 401.301990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 401.330216] alg: No test for adler32 (adler32-zlib) [ 402.362562] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 402.528184] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 404.183678] Key type lgssc registered [ 405.182941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 412.260362] loop: module loaded [ 412.306168] loop0: detected capacity change from 0 to 204800000 [ 412.499321] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 412.515902] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 412.777273] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 412.823774] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 417.130321] loop0: detected capacity change from 0 to 204800000 [ 417.300771] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 417.306738] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 417.313550] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 417.404726] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 421.265925] loop0: detected capacity change from 0 to 204800000 [ 425.331178] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 425.385861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 425.410643] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 426.613517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 426.649713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 426.714676] Lustre: lustre-MDT0000: new disk, initializing [ 426.790847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.808280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 426.831321] Lustre: 24283:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 466, rollback = 2 [ 426.836505] Lustre: 24283:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 426.840603] Lustre: 24283:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 2/466/0 [ 426.844500] Lustre: 24283:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 426.849594] Lustre: 24283:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 426.853389] Lustre: 24283:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 429.989321] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 440.463027] loop1: detected capacity change from 0 to 204800000 [ 450.483552] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 450.617428] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 451.271588] Lustre: lustre-OST07c6: new disk, initializing [ 451.274597] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 451.310055] Lustre: Skipped 1 previous similar message [ 451.447265] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 458.417981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 458.431066] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 458.475915] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 460.788829] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 466.033954] Lustre: 24308:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 466.047506] Lustre: 24308:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 2 previous similar messages [ 466.054681] Lustre: 24308:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 466.058872] Lustre: 24308:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 466.063422] Lustre: 24308:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 466.072864] Lustre: 24308:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 466.078129] Lustre: 24308:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 466.083534] Lustre: 24308:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 466.088327] Lustre: 24308:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 466.091584] Lustre: 24308:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 466.099259] Lustre: 24308:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 466.110440] Lustre: 24308:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 483.157287] Lustre: 24309:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 497, rollback = 2 [ 483.161940] Lustre: 24309:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 483.171249] Lustre: 24309:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 483.180189] Lustre: 24309:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 483.186545] Lustre: 24309:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 483.191148] Lustre: 24309:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 487.815238] Lustre: 24307:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 497, rollback = 2 [ 487.820541] Lustre: 24307:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 487.826699] Lustre: 24307:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 487.833901] Lustre: 24307:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 487.839569] Lustre: 24307:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 487.844971] Lustre: 24307:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 487.850598] Lustre: 24307:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 487.860531] Lustre: 24307:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 487.875842] Lustre: 24307:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 487.884384] Lustre: 24307:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 487.901183] Lustre: 24307:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 487.911173] Lustre: 24307:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 496.668835] Lustre: 24309:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 497, rollback = 2 [ 496.688197] Lustre: 24309:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 3 previous similar messages [ 496.703018] Lustre: 24309:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 496.714263] Lustre: 24309:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 496.728097] Lustre: 24309:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 496.732857] Lustre: 24309:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 496.737234] Lustre: 24309:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 496.742032] Lustre: 24309:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 496.746416] Lustre: 24309:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 496.750880] Lustre: 24309:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 496.753933] Lustre: 24309:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 496.757677] Lustre: 24309:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 507.708956] Lustre: 24309:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 497, rollback = 2 [ 507.718981] Lustre: 24309:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 5 previous similar messages [ 507.726094] Lustre: 24309:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 507.745450] Lustre: 24309:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 507.759149] Lustre: 24309:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 507.769997] Lustre: 24309:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 507.775683] Lustre: 24309:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 507.780792] Lustre: 24309:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 507.786367] Lustre: 24309:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 507.790682] Lustre: 24309:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 507.794315] Lustre: 24309:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 507.799081] Lustre: 24309:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 556.006178] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.008128] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 556.026951] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 561.120802] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 566.248985] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 569.311216] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 569.349746] LustreError: 26496:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 569.466685] Lustre: server umount lustre-OST07c6 complete [ 578.755881] LustreError: 26998:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 578.770937] LustreError: 26998:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 579.148952] Lustre: server umount lustre-MDT0000 complete [ 595.404628] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 598.053403] Key type lgssc unregistered [ 598.341142] LNet: 27831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 598.366524] LNetError: 27831:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 598.386626] LNet: Removed LNI 192.168.203.102@tcp [ 599.236253] Key type .llcrypt unregistered [ 599.244809] Key type ._llcrypt unregistered [ 616.251062] Key type ._llcrypt registered [ 616.252827] Key type .llcrypt registered [ 616.400737] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 617.394561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 617.454485] alg: No test for adler32 (adler32-zlib) [ 618.543724] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 618.744580] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 620.415331] Key type lgssc registered [ 621.540657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 629.172493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 629.206997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.869989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 634.933915] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 639.979913] LustreError: 29084:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.997916] LustreError: 29084:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 644.110450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 644.511615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 648.290846] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 653.795620] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 657.426444] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 664.447291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 664.979733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 670.331368] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 675.266544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 675.769844] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 681.695226] LustreError: 31283:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 690.659709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 690.660826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 690.677294] Lustre: Skipped 1 previous similar message [ 690.677880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 690.683509] Lustre: Skipped 1 previous similar message [ 695.784703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 695.795125] Lustre: Skipped 1 previous similar message [ 696.164639] LustreError: 31535:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 696.420311] Lustre: server umount lustre-OST0000 complete [ 700.898980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 700.900687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 700.914121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 700.943643] Lustre: Skipped 1 previous similar message [ 706.017721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 706.035015] Lustre: Skipped 2 previous similar messages [ 706.265575] LustreError: 31736:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 706.282745] LustreError: 31736:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 706.457355] Lustre: server umount lustre-MDT0000 complete [ 710.114995] LustreError: 29062:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761285014 with bad export cookie 7879491025920282014 [ 710.121358] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 710.144187] LustreError: 29062:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 710.350291] LustreError: 31940:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 710.353986] LustreError: 31940:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 710.565406] Lustre: server umount lustre-MDT0001 complete [ 721.676789] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 724.790507] Key type lgssc unregistered [ 725.174957] LNet: 32473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.179965] LNetError: 32473:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.200836] LNet: Removed LNI 192.168.203.102@tcp [ 726.012094] Key type .llcrypt unregistered [ 726.013627] Key type ._llcrypt unregistered [ 743.577931] Key type ._llcrypt registered [ 743.579458] Key type .llcrypt registered [ 743.681715] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 744.847523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 744.915822] alg: No test for adler32 (adler32-zlib) [ 746.041569] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 746.350700] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 748.104173] Key type lgssc registered [ 749.363738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 757.298722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 757.344174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 759.173091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 762.982819] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 769.505735] LustreError: 33723:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 769.514769] LustreError: 33723:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 771.407712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 771.936797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 775.632626] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 780.611578] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 784.308951] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 790.995245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 791.368330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 796.757767] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 802.757875] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 804.733510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 820.192575] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 820.208724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 820.220670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 820.230167] Lustre: Skipped 1 previous similar message [ 820.261390] Lustre: Skipped 1 previous similar message [ 823.332376] LustreError: 35995:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 823.656760] Lustre: server umount lustre-OST0000 complete [ 828.384579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 828.402560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 828.420375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 830.442520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 830.445845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 830.472094] Lustre: Skipped 1 previous similar message [ 833.499892] LustreError: 36196:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 833.513227] LustreError: 36196:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 833.690260] Lustre: server umount lustre-MDT0000 complete [ 835.553574] LustreError: 33723:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 835.585048] LustreError: 33723:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 837.556222] LustreError: 33704:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761285141 with bad export cookie 6600101237561485225 [ 837.560996] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 837.568187] LustreError: 33704:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 837.748144] LustreError: 36397:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 837.750603] LustreError: 36397:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 838.000557] Lustre: server umount lustre-MDT0001 complete [ 848.243350] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 851.099433] Key type lgssc unregistered [ 851.386063] LNet: 36930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.406402] LNetError: 36930:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.452814] LNet: Removed LNI 192.168.203.102@tcp [ 853.443866] Key type .llcrypt unregistered [ 853.455031] Key type ._llcrypt unregistered [ 876.500490] Key type ._llcrypt registered [ 876.502258] Key type .llcrypt registered [ 876.600722] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 877.564136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 877.578671] alg: No test for adler32 (adler32-zlib) [ 878.683880] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 878.865926] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 880.529134] Key type lgssc registered [ 881.847391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 889.700767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 889.743083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 891.369789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 895.087116] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 901.602398] LustreError: 38330:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 901.612140] LustreError: 38330:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 903.956414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 904.555615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 908.142989] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 913.337453] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 916.714696] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 922.353529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 922.690181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 927.770399] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 932.372509] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 935.987473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 943.946878] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.2@tcp (stopping) [ 945.633658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 945.637152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 945.650291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 946.148581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 949.050988] LustreError: 40582:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 949.065374] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.2@tcp (stopping) [ 949.084882] Lustre: Skipped 3 previous similar messages [ 949.187905] Lustre: server umount lustre-MDT0000 complete [ 951.269618] LustreError: 38326:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 951.282045] LustreError: 38326:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 953.327671] LustreError: 38311:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761285257 with bad export cookie 16716670535783266777 [ 953.332629] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 953.346302] LustreError: 38311:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 954.167915] LustreError: 38326:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 954.216615] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.2@tcp (stopping) [ 956.384804] LustreError: 38324:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.385823] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 956.472317] Lustre: Skipped 1 previous similar message [ 958.437485] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 958.440168] Lustre: Skipped 1 previous similar message [ 959.523071] LustreError: 40784:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 959.530654] LustreError: 40784:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 959.757070] Lustre: server umount lustre-MDT0001 complete [ 980.028260] LustreError: 41006:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 980.031415] LustreError: 41006:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 980.217478] Lustre: server umount lustre-OST0000 complete [ 995.831616] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 997.959783] Key type lgssc unregistered [ 998.215563] LNet: 41845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.224933] LNetError: 41845:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.239656] LNet: Removed LNI 192.168.203.102@tcp [ 999.130443] Key type .llcrypt unregistered [ 999.132057] Key type ._llcrypt unregistered [ 1016.949499] Key type ._llcrypt registered [ 1016.955248] Key type .llcrypt registered [ 1017.045116] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 1017.993341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1018.105606] alg: No test for adler32 (adler32-zlib) [ 1019.194696] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 1019.420484] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 1021.087707] Key type lgssc registered [ 1022.190640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1029.229873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1029.242078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1030.537281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1034.017205] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1035.749298] LustreError: 43222:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1035.766340] LustreError: 43222:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1040.870490] LustreError: 43222:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1041.034913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1041.287806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1044.140491] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1048.666369] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1052.067719] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1058.307162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1058.592677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1063.679219] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1067.985826] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1076.013940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1083.752430] Lustre: lustre-OST0000: Not available for connect from 192.168.203.2@tcp (stopping) [ 1086.434715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1086.439166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1086.440157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1086.453828] Lustre: Skipped 1 previous similar message [ 1088.714121] LustreError: 45492:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1088.948534] Lustre: server umount lustre-OST0000 complete [ 1106.790484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1106.796729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1106.799568] Lustre: Skipped 2 previous similar messages [ 1112.037702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1112.040577] Lustre: Skipped 1 previous similar message [ 1117.153629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1117.166130] Lustre: Skipped 1 previous similar message [ 1119.199135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1119.299543] LustreError: 45866:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1119.314841] LustreError: 45866:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1119.383210] Lustre: server umount lustre-MDT0000 complete [ 1122.040106] LustreError: 43202:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761285426 with bad export cookie 15853220841825377456 [ 1122.046823] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1122.061075] LustreError: 43202:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1128.211852] LustreError: 46069:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1128.214149] LustreError: 46069:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1128.387092] Lustre: server umount lustre-MDT0001 complete [ 1134.835976] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 1137.165620] Key type lgssc unregistered [ 1137.402145] LNet: 46602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1137.407982] LNetError: 46602:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1137.425738] LNet: Removed LNI 192.168.203.102@tcp [ 1138.173166] Key type .llcrypt unregistered [ 1138.176082] Key type ._llcrypt unregistered [ 1156.026306] Key type ._llcrypt registered [ 1156.028226] Key type .llcrypt registered [ 1156.091354] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 1156.766223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1156.779426] alg: No test for adler32 (adler32-zlib) [ 1157.777498] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 1157.940130] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 1159.567172] Key type lgssc registered [ 1160.492630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1167.287740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1167.305166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1168.701826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1171.701722] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1178.082802] LustreError: 47978:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1178.098281] LustreError: 47978:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1178.695634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1179.039749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1182.126890] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1185.789993] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1188.460195] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1194.146948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1194.441055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1198.535514] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1201.846234] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1213.872988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1222.322083] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1224.477202] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1236.025959] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-24 1h58m57s [ 1237.179673] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1239.388052] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.2@tcp (stopping) [ 1239.524114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.529489] Lustre: Skipped 2 previous similar messages [ 1240.439929] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.2@tcp (stopping) [ 1240.445654] Lustre: Skipped 3 previous similar messages [ 1244.809319] LustreError: 50567:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1244.858833] Lustre: server umount lustre-MDT0000 complete [ 1246.524873] LustreError: 47974:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.388907] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1251.578798] LustreError: 47974:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1251.609368] LustreError: 47974:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 1252.613945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1252.786882] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1253.005794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1253.080895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1254.071074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1256.499108] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1259.667866] Lustre: DEBUG MARKER: Wait for df (36003) ... [ 1261.019752] Lustre: DEBUG MARKER: done [ 1264.616791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.617867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1264.648639] Lustre: Skipped 1 previous similar message [ 1264.653782] Lustre: Skipped 3 previous similar messages [ 1269.732535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1269.742557] Lustre: Skipped 2 previous similar messages [ 1269.956030] LustreError: 51624:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1269.959694] LustreError: 51624:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1270.090961] Lustre: server umount lustre-OST0000 complete [ 1274.851972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1274.855248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1274.867183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1274.871322] Lustre: Skipped 1 previous similar message [ 1279.847639] LustreError: 51826:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1279.851272] LustreError: 51826:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1279.919003] Lustre: server umount lustre-MDT0000 complete [ 1280.995750] LustreError: 47974:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.010448] LustreError: 47974:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 1282.709551] LustreError: 47960:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761285586 with bad export cookie 17786436125474208557 [ 1282.715977] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1282.727768] LustreError: 47960:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1282.838489] LustreError: 52029:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1282.841741] LustreError: 52029:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1283.021573] Lustre: server umount lustre-MDT0001 complete [ 1290.505322] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 1292.645532] Key type lgssc unregistered [ 1292.922054] LNet: 52560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1292.936592] LNetError: 52560:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1292.949945] LNet: Removed LNI 192.168.203.102@tcp [ 1293.869207] Key type .llcrypt unregistered [ 1293.871219] Key type ._llcrypt unregistered [ 1325.218217] Key type ._llcrypt registered [ 1325.219433] Key type .llcrypt registered [ 1325.333577] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 1337.343895] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 1338.297717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1338.333254] alg: No test for adler32 (adler32-zlib) [ 1339.320840] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 1339.466396] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 1341.119190] Key type lgssc registered [ 1342.022033] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1349.637236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1354.431635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1359.862603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1365.478823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1371.713788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1371.766717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1371.782906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1373.047726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1373.083657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1373.146132] Lustre: lustre-MDT0000: new disk, initializing [ 1373.241587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.260394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1376.408872] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1385.108732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1385.186170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1385.314624] Lustre: 56504:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1385.402293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1385.406482] Lustre: Skipped 1 previous similar message [ 1385.507708] Lustre: lustre-MDT0001: new disk, initializing [ 1385.626936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1385.666253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1385.681700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1385.806818] Lustre: 57302:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 1385.827358] Lustre: 57302:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 1385.841736] Lustre: 57302:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 1385.855916] Lustre: 57302:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 1385.883094] Lustre: 57302:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 1385.916033] Lustre: 57302:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 1389.423128] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1395.484842] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1398.830695] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1404.891301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1404.960795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1405.180713] Lustre: lustre-OST0000: new disk, initializing [ 1405.184561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1405.270289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1409.844872] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1415.238697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1415.243970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1415.330993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1416.661313] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1422.035901] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1422.325850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1425.817288] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1425.986943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1431.013409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1431.014596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1431.019008] LustreError: Skipped 1 previous similar message [ 1431.019795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1431.023662] Lustre: Skipped 1 previous similar message [ 1436.139098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1436.149691] Lustre: Skipped 2 previous similar messages [ 1441.255980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1442.274308] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1442.321750] LustreError: 59446:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1442.588667] Lustre: server umount lustre-OST0000 complete [ 1446.368562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.383453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1446.387629] Lustre: Skipped 2 previous similar messages [ 1451.493970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1451.496770] Lustre: Skipped 1 previous similar message [ 1452.223622] LustreError: 59649:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1452.232214] LustreError: 59649:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1452.339537] Lustre: server umount lustre-MDT0000 complete [ 1455.759170] LustreError: 56495:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761285759 with bad export cookie 1951376058491747112 [ 1455.770413] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1455.773105] LustreError: 56495:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1455.864700] LustreError: 59851:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1455.917630] LustreError: 59851:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1456.239537] Lustre: server umount lustre-MDT0001 complete [ 1463.063515] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 02:02:45 (1761285765) [ 1467.661756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1468.030300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1471.051248] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1473.510071] LustreError: 60522:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1473.524788] LustreError: 60522:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 1477.727398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1477.929908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1480.997123] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1485.897095] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1489.046731] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1495.295958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1495.524962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1500.697442] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1505.407138] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1520.969391] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-24 2h03m43s [ 1522.423306] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1527.316735] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1529.017813] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1529.057955] LustreError: 60518:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1549.171688] Lustre: lustre-MDT0000: Client 370b4df5-920b-4cef-8d08-1d856926ad38 (at 192.168.203.2@tcp) reconnecting [ 1569.103167] LustreError: 60518:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1571.105214] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1573.264973] Lustre: DEBUG MARKER: done [ 1580.516804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1580.517080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1580.522124] Lustre: Skipped 1 previous similar message [ 1580.534977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1583.944243] LustreError: 63547:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1583.970831] LustreError: 63547:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1584.265559] Lustre: server umount lustre-OST0000 complete [ 1590.753283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1590.754465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1590.769359] LustreError: Skipped 1 previous similar message [ 1590.795517] Lustre: Skipped 2 previous similar messages [ 1594.334213] LustreError: 63748:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1594.343696] LustreError: 63748:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1594.499818] Lustre: server umount lustre-MDT0000 complete [ 1597.924241] LustreError: 63234:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1597.933932] LustreError: 63234:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 1598.188713] LustreError: 60501:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761285902 with bad export cookie 1951376058491748036 [ 1598.201202] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1598.211914] LustreError: 60501:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1598.606836] Lustre: server umount lustre-MDT0001 complete [ 1608.331557] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 1610.841043] Key type lgssc unregistered [ 1611.120582] LNet: 64484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1611.130822] LNetError: 64484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1611.144295] LNet: Removed LNI 192.168.203.102@tcp [ 1611.976898] Key type .llcrypt unregistered [ 1611.979742] Key type ._llcrypt unregistered [ 1641.053548] Key type ._llcrypt registered [ 1641.055175] Key type .llcrypt registered [ 1641.185302] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 1653.151291] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 1654.129839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1654.164793] alg: No test for adler32 (adler32-zlib) [ 1655.244617] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 1655.421754] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 1657.065150] Key type lgssc registered [ 1657.922713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1665.920044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1672.155637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1677.013769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1682.001855] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1687.087193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1687.159692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1687.182664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1688.417844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1688.455750] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1688.531181] Lustre: lustre-MDT0000: new disk, initializing [ 1688.656396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.687441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1691.839142] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1700.695099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1700.756138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1700.855203] Lustre: 68412:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1700.911214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1700.914796] Lustre: Skipped 1 previous similar message [ 1700.965588] Lustre: lustre-MDT0001: new disk, initializing [ 1701.043964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1701.077755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1701.089421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1704.112252] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1710.726277] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1714.473212] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1721.533696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1721.626662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1721.892249] Lustre: lustre-OST0000: new disk, initializing [ 1721.903788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1721.998840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1727.658533] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1729.586249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1729.593179] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1729.711870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1735.075334] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1738.978958] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1739.170333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1742.802292] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1742.999657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1745.378319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1745.380242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1745.387571] LustreError: Skipped 1 previous similar message [ 1745.395149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1745.402911] Lustre: Skipped 1 previous similar message [ 1750.496879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1750.502808] Lustre: Skipped 1 previous similar message [ 1750.708779] LustreError: 71353:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1750.892154] Lustre: server umount lustre-OST0000 complete [ 1755.617176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1755.626248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1755.637727] Lustre: Skipped 2 previous similar messages [ 1760.085050] LustreError: 71554:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1760.087531] LustreError: 71554:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1760.161903] Lustre: server umount lustre-MDT0000 complete [ 1763.294143] LustreError: 69194:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761286067 with bad export cookie 4518539160455094075 [ 1763.295886] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1763.302279] LustreError: 69194:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1763.420678] LustreError: 71758:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1763.431634] LustreError: 71758:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1763.707580] Lustre: server umount lustre-MDT0001 complete [ 1771.476751] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 02:07:53 (1761286073) [ 1772.942362] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1774.707535] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 02:07:56 (1761286076) [ 1785.283321] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 1788.788599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1789.206132] LustreError: 72807:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1789.226808] LustreError: 72807:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 1789.274115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1790.042822] Lustre: Failing over lustre-MDT0000 [ 1790.047211] LustreError: 72835:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1790.055328] LustreError: 72835:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1790.061904] LustreError: 72835:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1790.195767] LustreError: 72887:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1790.201264] LustreError: 72887:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1790.411958] Lustre: server umount lustre-MDT0000 complete [ 1795.841446] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:08:17 (1761286097) [ 1801.195256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1801.652386] LustreError: 73455:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1801.723341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1805.161451] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1806.816472] LustreError: 73455:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1810.917903] LustreError: 73455:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1811.766769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1812.080112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1815.325459] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1819.503798] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1822.220456] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1827.332355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1831.596673] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1836.302424] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1851.090749] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1856.487153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.492590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1856.496593] Lustre: Skipped 2 previous similar messages [ 1856.504415] Lustre: Skipped 2 previous similar messages [ 1857.773264] LustreError: 75820:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1857.782733] LustreError: 75820:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1859.917584] Lustre: server umount lustre-MDT0000 complete [ 1861.602246] LustreError: 73449:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1861.620502] LustreError: 73449:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1863.307074] LustreError: 73435:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761286167 with bad export cookie 4518539160455095265 [ 1863.318079] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1863.327451] LustreError: 73435:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1863.659752] Lustre: server umount lustre-MDT0001 complete [ 1866.620169] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1870.566257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1878.073631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1882.079151] Lustre: 66627:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761286170/real 1761286170] req@00000000c5d6859d x1846842185820032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761286186 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1882.097082] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.198449] Lustre: 66627:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761286172/real 1761286172] req@00000000fc96137e x1846842185820160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761286188 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1887.199436] Lustre: 66624:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761286175/real 1761286175] req@0000000078b732dd x1846842185820288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761286191 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1888.757347] LustreError: 75065:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1888.917708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1888.938136] Lustre: Skipped 1 previous similar message [ 1889.067447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1889.824131] Lustre: 66625:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761286177/real 1761286177] req@000000002b532ec3 x1846842185820544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761286193 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1893.085501] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1894.054847] Lustre: 66626:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761286182/real 1761286182] req@00000000baa3b5fa x1846842185820672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761286198 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1902.010115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1906.055816] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1907.689726] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1910.791397] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1911.820675] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1914.383970] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1922.023164] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.024360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1922.031078] Lustre: Skipped 2 previous similar messages [ 1922.041682] Lustre: Skipped 3 previous similar messages [ 1922.132089] LustreError: 78286:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1922.134778] LustreError: 78286:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1922.206909] Lustre: server umount lustre-MDT0000 complete [ 1925.291896] LustreError: 73436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761286229 with bad export cookie 4518539160455097176 [ 1925.296517] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1925.303069] LustreError: 73436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1925.634061] Lustre: server umount lustre-MDT0001 complete [ 1931.910389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1939.980366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.520144] Lustre: 66624:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761286231/real 1761286231] req@00000000ca8c3ef3 x1846842185844864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761286247 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1943.535190] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1950.951454] LustreError: 75065:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1950.959888] LustreError: 75065:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 1951.005504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1951.009086] Lustre: Skipped 1 previous similar message [ 1951.052905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1953.976141] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1961.521774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1962.923464] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1965.606264] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1970.907401] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1973.860556] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1981.419130] LustreError: 80745:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1981.431538] LustreError: 80745:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1981.526544] Lustre: server umount lustre-MDT0000 complete [ 1982.448461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1982.460192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.464200] LustreError: 79342:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1983.481334] LustreError: 79342:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 7 previous similar messages [ 1984.467555] LustreError: 76024:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761286288 with bad export cookie 4518539160455098548 [ 1984.470239] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1984.472762] LustreError: 76024:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1984.758488] Lustre: server umount lustre-MDT0001 complete [ 1989.993340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1993.187708] Lustre: DEBUG MARKER: files compared the same [ 2005.489261] Lustre: server umount lustre-OST0000 complete [ 2021.818206] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 2024.433792] Key type lgssc unregistered [ 2024.763148] LNet: 82377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2024.777150] LNetError: 82377:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2024.808849] LNet: Removed LNI 192.168.203.102@tcp [ 2025.645907] Key type .llcrypt unregistered [ 2025.647737] Key type ._llcrypt unregistered [ 2046.329461] Key type ._llcrypt registered [ 2046.331106] Key type .llcrypt registered [ 2046.440824] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 2047.238187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2047.417513] alg: No test for adler32 (adler32-zlib) [ 2048.494154] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 2048.633689] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 2050.306414] Key type lgssc registered [ 2051.414600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2058.918927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2058.938437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.413968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.348684] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug +malloc all [ 2069.989248] LustreError: 83757:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2070.022467] LustreError: 83757:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 2073.832916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2074.285246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2078.166356] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug +malloc all [ 2083.280158] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2086.181089] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2092.428401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2092.684592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.440677] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug +malloc all [ 2102.090912] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2106.029885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2115.552757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2115.556231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2115.566482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2116.580244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.581974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2116.589928] Lustre: Skipped 1 previous similar message [ 2119.674357] LustreError: 86004:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2120.026107] Lustre: server umount lustre-OST0000 complete [ 2125.796065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2125.814820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2125.840190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2130.094611] LustreError: 86206:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2130.097257] LustreError: 86206:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2130.229712] Lustre: server umount lustre-MDT0000 complete [ 2131.945642] LustreError: 83758:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.964060] LustreError: 83758:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2133.485935] LustreError: 83737:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761286437 with bad export cookie 16365790023432817273 [ 2133.490128] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2133.503489] LustreError: 83737:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2133.711795] LustreError: 86408:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2133.719933] LustreError: 86408:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2134.148464] Lustre: server umount lustre-MDT0001 complete [ 2145.013155] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 2147.680443] Key type lgssc unregistered [ 2147.936362] LNet: 86941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2147.941562] LNetError: 86941:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2147.977954] LNet: Removed LNI 192.168.203.102@tcp [ 2148.694247] Key type .llcrypt unregistered [ 2148.698086] Key type ._llcrypt unregistered [ 2168.732751] Key type ._llcrypt registered [ 2168.734464] Key type .llcrypt registered [ 2168.850339] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 2169.655193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2169.667470] alg: No test for adler32 (adler32-zlib) [ 2170.634370] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 2170.838975] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 2172.463945] Key type lgssc registered [ 2173.283651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2179.863524] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2179.877570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2195.487358] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2200.607197] LustreError: 88309:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.102@tcp: failed processing log, type 4: rc = -110 [ 2227.295182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2231.623618] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2235.824366] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2241.853281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.997734] Lustre: *** cfs_fail_loc=706, val=0*** [ 2242.413858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2242.486388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2246.085172] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2248.676639] LustreError: 89380:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.705397] LustreError: 89380:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 2252.970242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2253.232669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2256.250619] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2259.873150] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2262.025868] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2263.521425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2263.530139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2263.538915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2268.128306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2268.140946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2268.155456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2269.424290] LustreError: 90636:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2269.559503] Lustre: server umount lustre-OST0000 complete [ 2273.760180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2273.766321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.771853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2273.781296] Lustre: Skipped 1 previous similar message [ 2273.787797] Lustre: Skipped 1 previous similar message [ 2278.678281] LustreError: 90838:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2278.684967] LustreError: 90838:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2278.830799] Lustre: server umount lustre-MDT0000 complete [ 2278.896708] LustreError: 89287:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.913368] LustreError: 89287:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2282.238777] LustreError: 88317:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761286586 with bad export cookie 14874538590954316367 [ 2282.240093] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2282.255162] LustreError: 88317:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2282.410392] LustreError: 91039:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2282.418560] LustreError: 91039:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2282.686794] Lustre: server umount lustre-MDT0001 complete [ 2292.242561] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 2294.798560] Key type lgssc unregistered [ 2295.050628] LNet: 91572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2295.063910] LNetError: 91572:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2295.086552] LNet: Removed LNI 192.168.203.102@tcp [ 2295.821488] Key type .llcrypt unregistered [ 2295.823054] Key type ._llcrypt unregistered [ 2317.141929] Key type ._llcrypt registered [ 2317.144392] Key type .llcrypt registered [ 2317.234801] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 2317.802295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2317.904799] alg: No test for adler32 (adler32-zlib) [ 2318.955388] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 2319.158480] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 2320.791224] Key type lgssc registered [ 2321.755125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2328.767904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2328.788523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2333.116197] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2340.103983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2340.445821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2343.979468] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2345.443064] LustreError: 93616:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.452338] LustreError: 93616:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 2350.571081] LustreError: 93621:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2350.599279] LustreError: 93621:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2352.262454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2352.628021] LustreError: 93617:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.647237] LustreError: 93617:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 2352.667822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2357.729728] LustreError: 94350:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2357.753033] LustreError: 94350:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 2358.760480] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2362.848788] LustreError: 93621:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2362.856289] LustreError: 93621:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2365.599777] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2365.607461] Lustre: MGS: Client cb4d3626-6775-49cd-a6ec-fba440175d61 (at 0@lo) reconnecting [ 2365.804496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2365.878404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2369.015254] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2386.407057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2386.410779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2386.428708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2386.466195] LustreError: Skipped 1 previous similar message [ 2386.493772] Lustre: Skipped 1 previous similar message [ 2388.557189] LustreError: 95589:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2388.729908] Lustre: server umount lustre-OST0000 complete [ 2396.644786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2396.649200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.661909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2396.666927] Lustre: Skipped 2 previous similar messages [ 2396.672955] Lustre: Skipped 1 previous similar message [ 2401.763240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2406.367150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2406.438462] LustreError: 95792:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2406.458851] LustreError: 95792:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2406.504781] Lustre: server umount lustre-MDT0000 complete [ 2406.882645] LustreError: 94350:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.891482] LustreError: 94350:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 2409.803736] LustreError: 95996:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2409.816261] LustreError: 95996:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2410.090993] Lustre: server umount lustre-MDT0001 complete [ 2413.310844] LustreError: 96196:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2413.313718] LustreError: 96196:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2413.385888] Lustre: server umount MGS complete [ 2422.786042] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:18:44 (1761286724) [ 2447.318365] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 2454.840529] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 2464.182686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2469.002245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2473.624445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2479.108477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2484.896111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2484.938486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.320382] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2496.328291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2496.379383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2496.568164] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2496.610122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2496.674275] Lustre: lustre-MDT0001: new disk, initializing [ 2496.768131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2496.791232] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2496.796656] Lustre: 100713:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2496.801454] LustreError: 100713:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2496.820653] LustreError: 100715:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2496.830690] LustreError: 100715:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2497.320072] LustreError: 100715:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2497.331869] LustreError: 100715:0:(fld_handler.c:251:fld_server_lookup()) Skipped 4558 previous similar messages [ 2497.338355] LustreError: 100715:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2497.348152] LustreError: 100715:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 4558 previous similar messages [ 2498.320097] LustreError: 100715:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2498.324752] LustreError: 100715:0:(fld_handler.c:251:fld_server_lookup()) Skipped 8890 previous similar messages [ 2498.338198] LustreError: 100715:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2498.344446] LustreError: 100715:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 8994 previous similar messages [ 2499.952574] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2500.320061] LustreError: 100715:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2500.324907] LustreError: 100715:0:(fld_handler.c:251:fld_server_lookup()) Skipped 18875 previous similar messages [ 2500.338875] LustreError: 100715:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2500.343806] LustreError: 100715:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 18824 previous similar messages [ 2504.320254] LustreError: 100715:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2504.335685] LustreError: 100715:0:(fld_handler.c:251:fld_server_lookup()) Skipped 32493 previous similar messages [ 2504.343638] LustreError: 100715:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2504.349953] LustreError: 100715:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 32440 previous similar messages [ 2508.786894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2508.941232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2509.260905] Lustre: lustre-OST0000: new disk, initializing [ 2509.274879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2509.393666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2510.631841] Lustre: 101481:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000008a3522c0 x1846842882041728/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1761286830 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2512.320292] LustreError: 100715:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2512.324866] LustreError: 100715:0:(fld_handler.c:251:fld_server_lookup()) Skipped 57189 previous similar messages [ 2512.345826] LustreError: 100715:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2512.356295] LustreError: 100715:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 57277 previous similar messages [ 2515.364753] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2522.678883] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2528.324764] LustreError: 100715:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2528.333419] LustreError: 100715:0:(fld_handler.c:251:fld_server_lookup()) Skipped 93929 previous similar messages [ 2528.346474] LustreError: 100715:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2528.350753] LustreError: 100715:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 93852 previous similar messages [ 2529.047600] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2529.056469] Lustre: MGS: Client 61fe5587-1a29-48b6-a557-aeedd04042b3 (at 0@lo) reconnecting [ 2529.091398] Lustre: 99997:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2529.103785] LustreError: 99997:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2529.128848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2529.275685] Lustre: lustre-MDT0000: new disk, initializing [ 2529.468165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2529.507961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2529.526648] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2530.595853] Lustre: 102426:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000007acd73c5 x1846842882048384/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1761286850 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2533.500565] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2535.777628] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2538.475196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2538.484557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2539.682846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2539.698141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2539.824883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2564.065492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2564.065498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2564.066192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2564.087662] Lustre: Skipped 2 previous similar messages [ 2567.699138] LustreError: 102950:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2567.709696] LustreError: 102950:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2567.900155] Lustre: server umount lustre-OST0000 complete [ 2574.304496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2574.307566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.319564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2574.323735] Lustre: Skipped 2 previous similar messages [ 2577.262554] LustreError: 103153:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2577.283185] LustreError: 103153:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2577.305123] Lustre: server umount lustre-MDT0000 complete [ 2579.470421] LustreError: 100687:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.480326] LustreError: 100687:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2586.630728] Lustre: server umount lustre-MDT0001 complete [ 2589.794381] Lustre: server umount MGS complete [ 2599.031240] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:21:41 (1761286901) [ 2614.464286] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 2616.917307] Key type lgssc unregistered [ 2617.215384] LNet: 104892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2617.232367] LNetError: 104892:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2618.280480] LNet: Removed LNI 192.168.203.102@tcp [ 2618.991100] Key type .llcrypt unregistered [ 2618.992996] Key type ._llcrypt unregistered [ 2634.029447] Key type ._llcrypt registered [ 2634.031284] Key type .llcrypt registered [ 2634.166599] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 2634.944498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2634.974115] alg: No test for adler32 (adler32-zlib) [ 2635.976124] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 2636.198542] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 2637.840742] Key type lgssc registered [ 2639.018857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2645.410117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2645.430801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2647.041694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2650.270832] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2657.025754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2657.250379] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2657.347152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2660.338381] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2664.884796] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2668.062147] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2674.173227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2674.591933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2680.303186] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2685.734590] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2692.017691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2702.307405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2702.310681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2702.312127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2702.323275] Lustre: Skipped 1 previous similar message [ 2707.438201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2707.442416] Lustre: Skipped 2 previous similar messages [ 2708.048154] LustreError: 108364:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2708.281603] Lustre: server umount lustre-OST0000 complete [ 2712.548966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2712.561454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2712.569664] Lustre: Skipped 1 previous similar message [ 2717.668914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2717.884202] LustreError: 108565:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2717.890959] LustreError: 108565:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2718.032332] Lustre: server umount lustre-MDT0000 complete [ 2721.403089] LustreError: 108264:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287025 with bad export cookie 1694507062295547906 [ 2721.406289] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2721.414585] LustreError: 108264:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2721.537832] LustreError: 108766:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2721.543651] LustreError: 108766:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2721.745930] Lustre: server umount lustre-MDT0001 complete [ 2731.420564] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 2733.958476] Key type lgssc unregistered [ 2734.230557] LNet: 109299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2734.243392] LNetError: 109299:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2734.263300] LNet: Removed LNI 192.168.203.102@tcp [ 2735.025778] Key type .llcrypt unregistered [ 2735.028306] Key type ._llcrypt unregistered [ 2747.648206] Key type ._llcrypt registered [ 2747.651336] Key type .llcrypt registered [ 2747.786279] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 2748.650667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2748.734491] alg: No test for adler32 (adler32-zlib) [ 2749.723617] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 2749.935164] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 2751.599250] Key type lgssc registered [ 2752.792593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2758.096795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2758.100783] LustreError: 110330:0:(tgt_mount.c:2438:server_fill_super()) cfs_race id 716 sleeping [ 2758.140943] LustreError: 110344:0:(tgt_mount.c:2438:server_fill_super()) cfs_fail_race id 716 waking [ 2758.149145] LustreError: 110330:0:(tgt_mount.c:2438:server_fill_super()) cfs_fail_race id 716 awake: rc=4955 [ 2758.162782] LustreError: 110330:0:(tgt_mount.c:2450:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2758.169734] LustreError: 110330:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2758.200031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2759.898363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2766.537768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2766.733864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2769.901171] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2774.773258] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing lsmod [ 2777.263734] LustreError: 111512:0:(tgt_mount.c:2438:server_fill_super()) cfs_race id 716 sleeping [ 2777.350411] LustreError: 111523:0:(tgt_mount.c:2438:server_fill_super()) cfs_fail_race id 716 waking [ 2777.356580] LustreError: 111512:0:(tgt_mount.c:2438:server_fill_super()) cfs_fail_race id 716 awake: rc=4909 [ 2777.359842] LustreError: 111512:0:(tgt_mount.c:2450:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2777.382680] LustreError: 111512:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2777.386397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2777.843690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2780.624681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2780.628638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2780.675440] Lustre: Skipped 2 previous similar messages [ 2780.679724] Lustre: Skipped 2 previous similar messages [ 2785.765043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2785.769318] Lustre: Skipped 2 previous similar messages [ 2786.926837] LustreError: 111708:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2786.974841] Lustre: server umount lustre-MDT0000 complete [ 2789.982535] LustreError: 110362:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287094 with bad export cookie 9407842350735466933 [ 2789.987178] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2789.992738] LustreError: 110362:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2790.009181] Lustre: 110376:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.102@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2790.119125] LustreError: 111909:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2790.124614] LustreError: 111909:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2790.352497] Lustre: server umount lustre-MDT0001 complete [ 2803.073604] LustreError: 112110:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2803.086611] LustreError: 112110:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2803.206075] Lustre: server umount lustre-OST0000 complete [ 2810.515024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2810.786643] LustreError: 112588:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2810.799919] LustreError: 112588:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 2810.852585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2814.012237] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2815.970716] LustreError: 112588:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2820.082982] LustreError: 112588:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.896601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2822.321165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2825.495297] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2829.746327] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2832.701946] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2838.327316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2838.590029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2843.266716] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2847.885090] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2854.954881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2864.095970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2864.102450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2864.117877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2865.125807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2865.127587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2869.115789] LustreError: 114868:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2869.122273] LustreError: 114868:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2869.233164] Lustre: server umount lustre-OST0000 complete [ 2873.825073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2873.837308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2873.845104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2873.851503] Lustre: Skipped 2 previous similar messages [ 2877.804845] LustreError: 115070:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2877.809088] LustreError: 115070:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2877.884532] Lustre: server umount lustre-MDT0000 complete [ 2880.490533] LustreError: 113168:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2880.499761] LustreError: 113168:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2880.547815] LustreError: 112569:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287184 with bad export cookie 9407842350735467605 [ 2880.553236] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2880.560958] LustreError: 112569:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2880.899682] Lustre: server umount lustre-MDT0001 complete [ 2890.471298] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 2892.405445] Key type lgssc unregistered [ 2892.625280] LNet: 115803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2892.632085] LNetError: 115803:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2892.641172] LNet: Removed LNI 192.168.203.102@tcp [ 2893.253267] Key type .llcrypt unregistered [ 2893.255263] Key type ._llcrypt unregistered [ 2910.715074] Key type ._llcrypt registered [ 2910.719700] Key type .llcrypt registered [ 2910.807390] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 2911.397245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2911.516847] alg: No test for adler32 (adler32-zlib) [ 2912.494176] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 2912.663546] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 2914.303151] Key type lgssc registered [ 2915.085204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2920.394834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2920.414355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2921.769287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2924.962358] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2931.173350] LustreError: 117181:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2931.192672] LustreError: 117181:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 2931.299950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2931.624459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2934.940540] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2939.684248] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2943.112833] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2949.240446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2949.557298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2954.179290] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2958.526373] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2965.938300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 2973.098743] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2975.956277] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2979.793217] LustreError: 119551:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2980.149155] Lustre: server umount lustre-OST0000 complete [ 2986.467878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2986.501157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2991.589813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2991.594777] Lustre: Skipped 1 previous similar message [ 2996.705116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2996.719160] Lustre: Skipped 1 previous similar message [ 2997.663460] Lustre: 116492:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761287285/real 1761287285] req@0000000087900eed x1846843503836800/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761287301 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2997.709476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2997.724117] Lustre: Skipped 1 previous similar message [ 2998.751373] Lustre: 116492:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761287286/real 1761287286] req@00000000aa2d9f11 x1846843503837312/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761287302 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2999.263635] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2999.398245] LustreError: 119756:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2999.409518] LustreError: 119756:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2999.552198] Lustre: server umount lustre-MDT0000 complete [ 3001.889588] Lustre: 116492:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761287290/real 1761287290] req@000000003f1e6502 x1846843503838848/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761287306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3002.855976] LustreError: 117908:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3002.890920] LustreError: 117908:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 3003.671204] LustreError: 117162:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287307 with bad export cookie 6421136871613821800 [ 3003.682543] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3003.688936] LustreError: 117162:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3003.950709] LustreError: 119959:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3003.961176] LustreError: 119959:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3004.349975] Lustre: server umount lustre-MDT0001 complete [ 3014.060603] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 3016.294473] Key type lgssc unregistered [ 3016.547056] LNet: 120492:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3016.553564] LNetError: 120492:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3017.574291] LNet: Removed LNI 192.168.203.102@tcp [ 3018.229915] Key type .llcrypt unregistered [ 3018.231672] Key type ._llcrypt unregistered [ 3034.111227] Key type ._llcrypt registered [ 3034.113152] Key type .llcrypt registered [ 3034.217433] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 3034.889859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3035.035751] alg: No test for adler32 (adler32-zlib) [ 3036.236794] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 3036.503399] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 3038.151161] Key type lgssc registered [ 3039.118733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3046.572915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3046.594633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.284228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3052.091667] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3058.658447] LustreError: 121695:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3058.678659] LustreError: 121695:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 3060.796570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3061.127506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3064.814753] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3070.652698] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3073.936384] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3081.520108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3081.936319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.682844] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3093.001256] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3093.199812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3101.664072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3101.673079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.688902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3103.719870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3103.720824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.133152] LustreError: 123943:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3107.319371] Lustre: server umount lustre-OST0000 complete [ 3112.416626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3112.427448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3112.438227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3112.441246] Lustre: Skipped 1 previous similar message [ 3119.073942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3119.076179] Lustre: Skipped 2 previous similar messages [ 3124.193736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3124.201645] Lustre: Skipped 2 previous similar messages [ 3125.215150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3125.341591] LustreError: 124145:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3125.368191] LustreError: 124145:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3125.451839] Lustre: server umount lustre-MDT0000 complete [ 3128.291486] LustreError: 121695:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3128.325810] LustreError: 121695:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 3129.575580] LustreError: 121675:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287433 with bad export cookie 13613246960343386628 [ 3129.593540] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3129.611176] LustreError: 121675:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3129.818626] LustreError: 124347:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3129.821878] LustreError: 124347:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3130.214565] Lustre: server umount lustre-MDT0001 complete [ 3139.959918] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 3142.198496] Key type lgssc unregistered [ 3142.464219] LNet: 124880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3142.473715] LNetError: 124880:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3142.489199] LNet: Removed LNI 192.168.203.102@tcp [ 3143.240655] Key type .llcrypt unregistered [ 3143.245323] Key type ._llcrypt unregistered [ 3166.397817] Key type ._llcrypt registered [ 3166.399567] Key type .llcrypt registered [ 3166.549874] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 3167.312090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3167.437356] alg: No test for adler32 (adler32-zlib) [ 3168.491376] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 3168.731280] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 3170.431189] Key type lgssc registered [ 3171.650715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3178.985927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3179.019604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.403361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3183.911912] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3190.413502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3190.726315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3193.936855] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3198.221195] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3201.581395] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3207.051933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3207.439162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.065606] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3217.880545] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3225.903369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3234.137165] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3237.278940] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3237.282159] Lustre: Skipped 2 previous similar messages [ 3241.155919] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3241.165603] Lustre: Skipped 2 previous similar messages [ 3241.527966] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3241.563852] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3245.741166] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.2@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3245.752268] Lustre: Skipped 2 previous similar messages [ 3253.813342] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.2@tcp 192.168.0.[2,10]@tcp [ 3257.243287] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.2/24@tcp in log lustre-MDT0000 [ 3257.250835] Lustre: Skipped 2 previous similar messages [ 3260.062808] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.2/24@tcp [ 3260.080292] Lustre: Skipped 1 previous similar message [ 3268.065147] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3268.068694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3268.091412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3269.093277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3269.095397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3269.101375] LustreError: Skipped 3 previous similar messages [ 3269.155808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3272.165671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3272.168926] Lustre: Skipped 1 previous similar message [ 3273.739160] LustreError: 129823:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3273.914312] Lustre: server umount lustre-OST0000 complete [ 3277.280162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3277.284289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3277.300158] Lustre: Skipped 1 previous similar message [ 3277.315844] Lustre: Skipped 1 previous similar message [ 3282.400139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3283.215054] LustreError: 130024:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3283.217946] LustreError: 130024:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3283.303844] Lustre: server umount lustre-MDT0000 complete [ 3287.260104] LustreError: 126276:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287591 with bad export cookie 5349175774392739482 [ 3287.268406] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3287.274741] LustreError: 126276:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3287.408498] LustreError: 130226:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3287.414692] LustreError: 130226:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3287.852475] Lustre: server umount lustre-MDT0001 complete [ 3297.868355] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 3300.239580] Key type lgssc unregistered [ 3300.462328] LNet: 130758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3300.469701] LNetError: 130758:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3300.484864] LNet: Removed LNI 192.168.203.102@tcp [ 3301.114874] Key type .llcrypt unregistered [ 3301.119757] Key type ._llcrypt unregistered [ 3328.025488] Key type ._llcrypt registered [ 3328.029103] Key type .llcrypt registered [ 3328.096897] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 3329.502491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3329.509642] alg: No test for adler32 (adler32-zlib) [ 3330.556358] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 3330.768565] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 3332.474095] Key type lgssc registered [ 3333.661452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3341.450082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3341.478538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.890175] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3342.911911] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.2/24@tcp [ 3342.913118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3346.887042] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3353.057615] LustreError: 132297:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3353.084753] LustreError: 132297:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 3354.768850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3355.054743] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3355.070891] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.2/24@tcp [ 3355.130242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3358.993537] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3363.468905] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3366.604615] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3373.075899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3373.459244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3378.526649] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3383.736983] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3388.841094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3399.138907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3399.141760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3399.152775] Lustre: Skipped 1 previous similar message [ 3399.160745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3404.263404] LustreError: 133904:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.282112] LustreError: 134608:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3404.293902] LustreError: 133904:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 3404.591639] Lustre: server umount lustre-OST0000 complete [ 3409.377856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.385891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3409.398247] Lustre: Skipped 1 previous similar message [ 3409.422417] Lustre: Skipped 2 previous similar messages [ 3414.497246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3414.507493] Lustre: Skipped 1 previous similar message [ 3419.616428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3419.626817] Lustre: Skipped 1 previous similar message [ 3422.175138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3422.306429] LustreError: 134809:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3422.309346] LustreError: 134809:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3422.544836] Lustre: server umount lustre-MDT0000 complete [ 3424.738094] LustreError: 133018:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3424.765885] LustreError: 133018:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 3426.343294] LustreError: 134810:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287730 with bad export cookie 4975560072510721181 [ 3426.343922] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3426.357603] LustreError: 134810:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3426.485605] LustreError: 135013:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3426.488819] LustreError: 135013:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3426.832756] Lustre: server umount lustre-MDT0001 complete [ 3438.041300] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 3441.034322] Key type lgssc unregistered [ 3441.354590] LNet: 135545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3441.362287] LNetError: 135545:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3441.381630] LNet: Removed LNI 192.168.203.102@tcp [ 3442.214453] Key type .llcrypt unregistered [ 3442.216610] Key type ._llcrypt unregistered [ 3467.651240] Key type ._llcrypt registered [ 3467.652923] Key type .llcrypt registered [ 3467.743685] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 3481.788794] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 3482.667422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3482.812122] alg: No test for adler32 (adler32-zlib) [ 3483.930857] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 3484.139881] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 3485.815182] Key type lgssc registered [ 3486.836439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3495.905287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3502.229703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3508.483942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3514.983699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3521.515655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3521.573103] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3521.604133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3522.850295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3522.884857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3522.980758] Lustre: lustre-MDT0000: new disk, initializing [ 3523.071222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3523.088567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3526.134592] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3534.748304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3534.830693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3534.951470] Lustre: 138924:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3535.033313] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3535.047159] Lustre: Skipped 1 previous similar message [ 3535.110138] Lustre: lustre-MDT0001: new disk, initializing [ 3535.204358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3535.235135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3535.252419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3538.676362] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3545.370680] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3548.546634] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3554.591364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3554.666051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3554.943672] Lustre: lustre-OST0000: new disk, initializing [ 3554.946711] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3555.045739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3560.329545] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3564.576369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3564.590336] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3564.684107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3567.153712] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3570.732660] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3570.953224] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3574.629673] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3574.891691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3579.875958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3579.880145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3579.888489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3579.892865] Lustre: Skipped 1 previous similar message [ 3582.785948] LustreError: 141863:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3583.023103] Lustre: server umount lustre-OST0000 complete [ 3586.530616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3586.539381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3586.567075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3586.579265] Lustre: Skipped 1 previous similar message [ 3590.116624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3590.118585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3595.234178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3595.242036] Lustre: Skipped 2 previous similar messages [ 3600.351157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3600.352859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3600.367699] Lustre: Skipped 1 previous similar message [ 3600.469285] LustreError: 142064:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3600.480639] LustreError: 142064:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3600.645325] Lustre: server umount lustre-MDT0000 complete [ 3603.970082] LustreError: 138914:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287908 with bad export cookie 10380142885510836157 [ 3603.976175] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3604.000574] LustreError: 138914:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3604.204934] LustreError: 142268:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3604.207261] LustreError: 142268:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3604.376592] Lustre: server umount lustre-MDT0001 complete [ 3611.924977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3612.240193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3616.169585] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3617.261527] LustreError: 142745:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3617.296668] LustreError: 142745:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 3622.374754] LustreError: 142745:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3624.823800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3625.117787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3629.632908] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3635.353506] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3639.399258] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3646.100361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3646.395697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3652.308738] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3657.830771] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3661.400394] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3661.690376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3665.037870] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3665.223242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3672.853476] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3672.937948] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3673.128368] Lustre: lustre-OST0001: new disk, initializing [ 3673.134809] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3673.225428] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3678.480132] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3681.385868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3681.399399] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3681.477249] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3685.244259] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3685.454580] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3688.846194] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3689.072701] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3692.528022] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c0bd0787000.ost_server_uuid 50 [ 3693.756345] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9c0bd0787000.ost_server_uuid in FULL state after 0 sec [ 3701.311855] LustreError: 146997:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3701.321759] LustreError: 146997:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3701.409463] Lustre: server umount lustre-OST0001 complete [ 3702.239954] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3702.245594] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3702.267319] LustreError: 144568:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3702.283875] Lustre: Skipped 1 previous similar message [ 3706.850330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3706.862405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3706.872732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3707.365595] LustreError: 144568:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3707.376477] LustreError: 144568:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 3710.528114] LustreError: 147197:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3710.530284] LustreError: 147197:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3710.647535] Lustre: server umount lustre-OST0000 complete [ 3717.603688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3717.604719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3717.620069] Lustre: Skipped 2 previous similar messages [ 3719.808945] LustreError: 147399:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3719.815547] LustreError: 147399:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3719.914712] Lustre: server umount lustre-MDT0000 complete [ 3722.727149] LustreError: 142742:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3722.738291] LustreError: 142742:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 3722.890403] LustreError: 142727:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288027 with bad export cookie 10380142885510837081 [ 3722.891851] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3722.896799] LustreError: 142727:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3723.216301] Lustre: server umount lustre-MDT0001 complete [ 3737.632547] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 3739.572508] Key type lgssc unregistered [ 3739.766345] LNet: 148588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3739.773494] LNetError: 148588:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3739.789058] LNet: Removed LNI 192.168.203.102@tcp [ 3740.449915] Key type .llcrypt unregistered [ 3740.452670] Key type ._llcrypt unregistered [ 3761.077550] Key type ._llcrypt registered [ 3761.079390] Key type .llcrypt registered [ 3761.145408] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 3770.386103] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 3771.082330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3771.111480] alg: No test for adler32 (adler32-zlib) [ 3772.119285] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 3772.268395] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 3773.879206] Key type lgssc registered [ 3774.565356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3780.412886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3784.606173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3788.502380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3792.903601] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3797.360177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3797.403136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3797.424545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3798.588325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3798.620751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3798.673744] Lustre: lustre-MDT0000: new disk, initializing [ 3798.737663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3798.758668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3801.394143] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3809.124873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3809.170353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3809.235752] Lustre: 152515:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3809.295535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3809.297661] Lustre: Skipped 1 previous similar message [ 3809.354000] Lustre: lustre-MDT0001: new disk, initializing [ 3809.435279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3809.469264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3809.476445] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3809.555224] Lustre: 153313:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 3809.566314] Lustre: 153313:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 3809.569843] Lustre: 153313:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 3809.578217] Lustre: 153313:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 3809.592634] Lustre: 153313:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 3809.599811] Lustre: 153313:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 3812.313430] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3817.829502] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3820.632280] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3824.947988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3824.996126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3825.187604] Lustre: lustre-OST0000: new disk, initializing [ 3825.194132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3825.257490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3829.029626] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3834.338289] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3834.915099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3834.920731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3834.996991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3836.957309] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3837.100154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3839.831624] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3840.036884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3845.094558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3845.095941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3845.097705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3845.097815] LustreError: Skipped 1 previous similar message [ 3845.121160] Lustre: Skipped 1 previous similar message [ 3847.552074] LustreError: 155457:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3847.690945] Lustre: server umount lustre-OST0000 complete [ 3850.721845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3850.736789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.753673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3850.757269] Lustre: Skipped 1 previous similar message [ 3855.356793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3855.359996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3856.630724] LustreError: 155659:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3856.636509] LustreError: 155659:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3858.721377] LustreError: 155659:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 3858.723995] LustreError: 155659:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3858.763701] Lustre: server umount lustre-MDT0000 complete [ 3861.539057] LustreError: 152505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288165 with bad export cookie 9610932907538353751 [ 3861.540176] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3861.558219] LustreError: 152505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3861.682415] LustreError: 155860:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3861.685805] LustreError: 155860:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3861.873392] Lustre: server umount lustre-MDT0001 complete [ 3868.191444] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 02:42:50 (1761288170) [ 3879.767581] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 3885.759444] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 3892.047732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3896.920938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3901.441496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3905.380967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3909.585566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3909.625754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3909.775850] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3909.804527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3909.853394] Lustre: lustre-MDT0000: new disk, initializing [ 3909.911663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3909.931935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3912.051935] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3918.753950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3918.795465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3918.870289] Lustre: 158848:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3918.917516] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3918.921771] Lustre: Skipped 1 previous similar message [ 3918.982984] Lustre: lustre-MDT0001: new disk, initializing [ 3919.057542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3919.090880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3919.100249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3919.151260] Lustre: 159646:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 3919.160438] Lustre: 159646:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 3919.164774] Lustre: 159646:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 3919.170032] Lustre: 159646:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3919.178724] Lustre: 159646:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 3919.184834] Lustre: 159646:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3919.188934] Lustre: 159646:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 3919.195320] Lustre: 159646:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3919.199770] Lustre: 159646:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 3919.202930] Lustre: 159646:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3919.210223] Lustre: 159646:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 3919.217492] Lustre: 159646:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3921.394958] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3926.202721] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3928.062331] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3931.998744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3932.037757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3932.186608] Lustre: lustre-OST0000: new disk, initializing [ 3932.190466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3932.241827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3935.384168] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3939.648899] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3940.363698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3940.367814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3940.418398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3947.750067] Lustre: Failing over lustre-OST0000 [ 3947.787190] LustreError: 161290:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3947.795415] LustreError: 161290:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3947.901885] Lustre: server umount lustre-OST0000 complete [ 3962.333852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3962.496255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3962.504441] Lustre: Skipped 1 previous similar message [ 3962.511163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3962.523676] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3965.367191] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3965.920152] Lustre: 150728:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761288254/real 1761288254] req@0000000043bbb381 x1846844405649536/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761288270 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3966.434174] Lustre: 150730:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761288254/real 1761288254] req@00000000d47eebe2 x1846844405650304/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761288270 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3966.670318] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3967.344670] Lustre: Failing over lustre-MDT0000 [ 3967.433022] LustreError: 162263:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3967.435955] LustreError: 162263:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3967.481554] Lustre: server umount lustre-MDT0000 complete [ 3967.972129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3967.983067] LustreError: 158857:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3967.991764] LustreError: 158857:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 3971.039234] Lustre: 150730:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761288259/real 1761288259] req@000000003d3f93e2 x1846844405651328/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761288275 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3971.050838] Lustre: 150730:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3971.908605] LustreError: 158856:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3973.088821] LustreError: 158860:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3973.097987] LustreError: 158860:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 3976.159221] Lustre: 150729:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761288264/real 1761288264] req@00000000eed71bb7 x1846844405652352/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761288280 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3977.016922] LustreError: 158857:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3982.136640] LustreError: 158857:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3982.149137] LustreError: 158857:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 3982.787946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3982.894197] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3982.899978] Lustre: 158852:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.102@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3983.037017] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3983.112650] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3984.204938] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3985.549838] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3988.460050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3988.473247] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3988.499716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3993.570504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3993.571201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3993.574923] LustreError: Skipped 1 previous similar message [ 3993.576537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3993.576542] Lustre: Skipped 1 previous similar message [ 3993.597677] Lustre: Skipped 3 previous similar messages [ 3996.472147] LustreError: 163148:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3996.474348] LustreError: 163148:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3996.597334] Lustre: server umount lustre-OST0000 complete [ 4003.811428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4003.812818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4003.818306] Lustre: Skipped 2 previous similar messages [ 4005.057627] Lustre: server umount lustre-MDT0000 complete [ 4007.337405] LustreError: 160596:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288311 with bad export cookie 9610932907538355921 [ 4007.342229] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4007.634271] Lustre: server umount lustre-MDT0001 complete [ 4014.678854] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 4016.316334] Key type lgssc unregistered [ 4016.480447] LNet: 164083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4016.485594] LNetError: 164083:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4016.494666] LNet: Removed LNI 192.168.203.102@tcp [ 4016.964602] Key type .llcrypt unregistered [ 4016.966362] Key type ._llcrypt unregistered [ 4042.657381] Key type ._llcrypt registered [ 4042.658855] Key type .llcrypt registered [ 4042.720908] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 4043.362421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4043.376769] alg: No test for adler32 (adler32-zlib) [ 4044.394084] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 4044.533720] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 4046.151282] Key type lgssc registered [ 4046.834212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4051.368375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4051.378629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4052.622967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4054.587587] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4058.654532] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4061.035220] LustreError: 165611:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4061.041463] LustreError: 165611:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4061.051082] LustreError: 165611:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 4061.135849] LustreError: 166136:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4061.258992] Lustre: server umount lustre-MDT0000 complete [ 4069.644293] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 4074.909505] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 4080.590940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4085.447761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4094.026146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4102.594240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4111.419345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4111.510431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4111.858492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4111.921169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4112.093377] Lustre: lustre-MDT0000: new disk, initializing [ 4112.218367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4112.249187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4117.860371] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4131.217475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4131.339584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4131.548226] Lustre: 168779:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4131.658834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4131.666669] Lustre: Skipped 1 previous similar message [ 4131.851744] Lustre: lustre-MDT0001: new disk, initializing [ 4131.998249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4132.061510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4132.072892] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4132.250505] Lustre: 169578:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 4132.274166] Lustre: 169578:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 4132.300793] Lustre: 169578:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 4132.318622] Lustre: 169578:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 4132.346948] Lustre: 169578:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 4132.362615] Lustre: 169578:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 4137.015699] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4144.740388] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4148.642936] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4155.061725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4155.133271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4155.336648] Lustre: lustre-OST0000: new disk, initializing [ 4155.342779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4155.424830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4160.788068] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4163.140602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4163.154693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4163.235994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4166.965065] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4171.275910] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4172.645104] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:47:54 (1761288474) [ 4387.807979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4387.811920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4387.854697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4388.834376] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4388.834884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4388.846243] Lustre: Skipped 1 previous similar message [ 4388.851199] Lustre: Skipped 1 previous similar message [ 4393.953569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4399.071626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4399.074362] Lustre: Skipped 4 previous similar messages [ 4401.119980] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4401.205595] LustreError: 172311:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4401.213955] LustreError: 172311:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4401.295284] Lustre: server umount lustre-MDT0000 complete [ 4404.176782] LustreError: 168772:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288708 with bad export cookie 10830904634728412952 [ 4404.181875] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4404.186932] LustreError: 168772:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4404.198303] LustreError: 172172:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.201840] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4404.225491] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4404.215631] LustreError: 172172:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 4409.312293] LustreError: 172172:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4410.407227] LustreError: 172513:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4410.418090] LustreError: 172513:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4410.641626] Lustre: server umount lustre-MDT0001 complete [ 4428.255124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4428.295548] LustreError: 172714:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4428.337746] LustreError: 172714:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4428.548692] Lustre: server umount lustre-OST0000 complete [ 4435.744759] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 4441.916414] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 4448.633989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4452.652255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4456.623580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4461.586421] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4465.573774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4465.616544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4465.759827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4465.794111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4465.838680] Lustre: lustre-MDT0000: new disk, initializing [ 4465.883917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4465.895921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4468.009257] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4475.729859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4475.773573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4475.828564] Lustre: 175057:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4475.876082] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4475.879555] Lustre: Skipped 1 previous similar message [ 4475.942364] Lustre: lustre-MDT0001: new disk, initializing [ 4476.022491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4476.063066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4476.075404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4476.140032] Lustre: 175851:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 4476.147595] Lustre: 175851:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 4476.155883] Lustre: 175851:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 4476.163251] Lustre: 175851:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4476.166433] Lustre: 175851:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 4476.174642] Lustre: 175851:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4476.180463] Lustre: 175851:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 4476.185754] Lustre: 175851:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4476.189544] Lustre: 175851:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 4476.194053] Lustre: 175851:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4476.197289] Lustre: 175851:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 4476.200505] Lustre: 175851:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4478.608638] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4483.557438] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4485.926272] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4491.162704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4491.239587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4491.410426] Lustre: lustre-OST0000: new disk, initializing [ 4491.413923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4494.776139] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4497.927120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4497.931587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4497.979769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4498.606061] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4500.391751] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4500.527683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4502.278678] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4502.373241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4508.131406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4508.132812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4508.134715] LustreError: Skipped 1 previous similar message [ 4508.135948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4508.140369] Lustre: Skipped 1 previous similar message [ 4508.142567] Lustre: Skipped 1 previous similar message [ 4509.232147] LustreError: 177996:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4509.234418] LustreError: 177996:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4509.322443] Lustre: server umount lustre-OST0000 complete [ 4511.713876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4513.251887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4513.256232] Lustre: Skipped 1 previous similar message [ 4517.266660] LustreError: 178198:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4517.270063] LustreError: 178198:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4517.328612] Lustre: server umount lustre-MDT0000 complete [ 4518.368410] LustreError: 175854:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4518.374916] LustreError: 175854:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 4519.084241] LustreError: 175048:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288823 with bad export cookie 10830904634728571824 [ 4519.087952] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4519.089932] LustreError: 175048:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4519.312334] Lustre: server umount lustre-MDT0001 complete [ 4522.909945] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:53:45 (1761288825) [ 4529.714124] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 4533.518840] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 4537.782864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4540.498107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4543.116318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4546.389319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4549.621996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4549.661191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4549.778185] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4549.796261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4549.841810] Lustre: lustre-MDT0000: new disk, initializing [ 4549.890810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4549.894700] Lustre: Skipped 1 previous similar message [ 4549.904364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4551.715931] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4556.861728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4556.894757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4556.942055] Lustre: 181383:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4557.074245] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4557.110880] Lustre: 182188:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 4557.116487] Lustre: 182188:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 4557.121220] Lustre: 182188:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 4557.126026] Lustre: 182188:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4557.129328] Lustre: 182188:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 4557.132667] Lustre: 182188:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4557.135805] Lustre: 182188:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 4557.141423] Lustre: 182188:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4557.147176] Lustre: 182188:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 4557.151679] Lustre: 182188:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4557.154933] Lustre: 182188:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 4557.158242] Lustre: 182188:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4558.794931] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4562.785413] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4564.452194] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4567.619383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4567.657352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4570.406629] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4574.224916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4574.279345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4574.340486] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4579.298161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4579.300902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4579.308336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4579.311028] Lustre: Skipped 5 previous similar messages [ 4583.226096] LustreError: 183935:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4583.228158] LustreError: 183935:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4583.307534] Lustre: server umount lustre-OST0000 complete [ 4588.002056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4588.005724] LustreError: Skipped 1 previous similar message [ 4599.263148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4599.339525] Lustre: server umount lustre-MDT0000 complete [ 4599.776852] LustreError: 181395:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4599.784160] LustreError: 181395:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 4600.929244] LustreError: 181376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288905 with bad export cookie 10830904634728572748 [ 4600.931655] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4600.934693] LustreError: 181376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4601.108772] Lustre: server umount lustre-MDT0001 complete [ 4604.622837] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:55:07 (1761288907) [ 4611.389524] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 4615.330504] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 4619.680639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4622.355507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4625.154341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4628.186829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4631.603060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4631.638082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4631.753039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4631.777306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4631.781454] Lustre: Skipped 3 previous similar messages [ 4631.836170] Lustre: lustre-MDT0000: new disk, initializing [ 4631.837696] Lustre: Skipped 2 previous similar messages [ 4631.889641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4631.893167] Lustre: Skipped 2 previous similar messages [ 4631.904750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4631.908317] Lustre: Skipped 2 previous similar messages [ 4633.794027] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4639.450694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4639.501894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4639.569681] Lustre: 187325:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4639.730259] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4639.764959] Lustre: 188125:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 4639.769714] Lustre: 188125:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 4639.773993] Lustre: 188125:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 4639.779854] Lustre: 188125:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4639.785806] Lustre: 188125:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 4639.791290] Lustre: 188125:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4639.795630] Lustre: 188125:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 4639.802970] Lustre: 188125:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4639.809655] Lustre: 188125:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 4639.815654] Lustre: 188125:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4639.819369] Lustre: 188125:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 4639.822731] Lustre: 188125:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4641.925305] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4646.088192] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4647.854044] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4650.884291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4650.915589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4653.456594] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4657.187679] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4660.822571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4665.829434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4665.831633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4665.834722] LustreError: Skipped 1 previous similar message [ 4665.846701] Lustre: Skipped 3 previous similar messages [ 4665.847552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4665.847558] Lustre: Skipped 5 previous similar messages [ 4667.315245] LustreError: 189871:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4667.322148] LustreError: 189871:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4667.399234] Lustre: server umount lustre-OST0000 complete [ 4675.461546] Lustre: server umount lustre-MDT0000 complete [ 4676.065765] LustreError: 188462:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4676.076780] LustreError: 188462:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 4677.502394] LustreError: 187315:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288981 with bad export cookie 10830904634728573875 [ 4677.505508] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4677.508037] LustreError: 187315:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4677.737285] Lustre: server umount lustre-MDT0001 complete [ 4682.752491] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 4684.117476] Key type lgssc unregistered [ 4684.280478] LNet: 190804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4684.284371] LNetError: 190804:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4684.297394] LNet: Removed LNI 192.168.203.102@tcp [ 4684.736640] Key type .llcrypt unregistered [ 4684.738738] Key type ._llcrypt unregistered [ 4696.961504] Key type ._llcrypt registered [ 4696.962734] Key type .llcrypt registered [ 4697.014973] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 4697.524895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4697.536448] alg: No test for adler32 (adler32-zlib) [ 4698.438462] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 4698.550787] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 4700.151175] Key type lgssc registered [ 4700.709943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4704.184222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4704.196221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4705.450530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4707.243933] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4711.091264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4711.291968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4713.053282] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4715.726082] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4717.415585] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4720.218516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4720.396324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4722.700815] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4724.812157] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4730.673427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4751.329136] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4751.330445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4751.334525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4751.340629] Lustre: Skipped 1 previous similar message [ 4753.593437] LustreError: 194451:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4753.711450] Lustre: server umount lustre-OST0000 complete [ 4756.448759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4756.450117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4756.454274] Lustre: Skipped 1 previous similar message [ 4756.458713] Lustre: Skipped 2 previous similar messages [ 4761.542735] LustreError: 194653:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4761.544863] LustreError: 194653:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4761.568435] LustreError: 194351:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4761.574861] LustreError: 194351:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 4761.581723] Lustre: server umount lustre-MDT0000 complete [ 4763.233392] LustreError: 194452:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289067 with bad export cookie 169420934903906166 [ 4763.236809] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4763.238766] LustreError: 194452:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4763.314233] LustreError: 194855:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4763.316688] LustreError: 194855:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4763.418113] Lustre: server umount lustre-MDT0001 complete [ 4767.821783] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 4769.039665] Key type lgssc unregistered [ 4769.186608] LNet: 195386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4769.190716] LNetError: 195386:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4769.199668] LNet: Removed LNI 192.168.203.102@tcp [ 4769.581482] Key type .llcrypt unregistered [ 4769.583546] Key type ._llcrypt unregistered [ 4780.576681] Key type ._llcrypt registered [ 4780.578254] Key type .llcrypt registered [ 4780.631202] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 4780.976146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4780.985174] alg: No test for adler32 (adler32-zlib) [ 4781.912478] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 4782.057370] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 4783.663222] Key type lgssc registered [ 4784.400082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4788.563291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4788.579828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4789.861191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4791.923973] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4796.827635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4797.037487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4799.222548] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4802.004884] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4803.585991] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4806.355660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4806.524885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4809.049723] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4811.500205] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4814.766627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4819.938303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4819.939140] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4819.944783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4819.949721] Lustre: Skipped 1 previous similar message [ 4822.347767] Lustre: lustre-OST0000: Not available for connect from 192.168.203.2@tcp (stopping) [ 4825.058334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4825.061229] Lustre: Skipped 1 previous similar message [ 4825.654157] LustreError: 199007:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4825.746515] Lustre: server umount lustre-OST0000 complete [ 4828.231966] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9c0bc9956800.ost_server_uuid 50 [ 4828.952848] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4839.390956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4839.392933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4839.396150] Lustre: Skipped 2 previous similar messages [ 4844.512527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4844.515336] Lustre: Skipped 2 previous similar messages [ 4845.522492] LustreError: 199357:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4845.524741] LustreError: 199357:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4845.586609] Lustre: server umount lustre-MDT0000 complete [ 4847.552804] LustreError: 196742:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289151 with bad export cookie 5111080691324140587 [ 4847.556548] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4847.562228] LustreError: 196742:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4847.650907] LustreError: 199560:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4847.657254] LustreError: 199560:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4847.767826] Lustre: server umount lustre-MDT0001 complete [ 4851.881486] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:59:14 (1761289154) [ 4855.048450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4855.314103] LustreError: 200231:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4855.327346] LustreError: 200231:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 4855.355813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4857.128835] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4860.385592] LustreError: 200231:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4861.366749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4861.602396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4863.450233] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4865.837131] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4867.579173] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4870.727133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4870.913541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4873.660262] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4876.168459] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4879.541754] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4879.571261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4879.652771] Lustre: lustre-OST0001: new disk, initializing [ 4879.655661] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4880.747449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4882.383498] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4884.527123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4884.532065] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4884.593611] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4886.251085] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4887.520273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4887.523551] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4887.529786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4889.570975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.425316] LustreError: 203440:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4893.428514] LustreError: 203440:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4893.454581] Lustre: server umount lustre-OST0000 complete [ 4894.690603] LustreError: 201828:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4894.700369] LustreError: 201828:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 4895.930696] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4896.045456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4897.080092] LustreError: 201828:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4897.703159] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4897.799719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4902.198875] LustreError: 203218:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4902.210097] LustreError: 203218:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 4910.049116] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4910.051738] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4910.056053] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4910.059492] Lustre: Skipped 1 previous similar message [ 4910.062567] Lustre: Skipped 3 previous similar messages [ 4914.614091] LustreError: 204141:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4914.617096] LustreError: 204141:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4914.701919] Lustre: server umount lustre-OST0001 complete [ 4918.240431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4918.243508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4922.639536] LustreError: 204343:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4922.642233] LustreError: 204343:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4922.690634] Lustre: server umount lustre-MDT0000 complete [ 4924.594448] LustreError: 202740:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289228 with bad export cookie 5111080691324141623 [ 4924.595465] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4924.600602] LustreError: 202740:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4924.778752] Lustre: server umount lustre-MDT0001 complete [ 4927.398226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4928.979769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4930.689160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4932.336485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4935.043266] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:00:37 (1761289237) [ 4937.881829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4938.017784] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4938.033884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4938.136562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4938.140153] Lustre: Skipped 1 previous similar message [ 4939.797111] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4943.205195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4943.257595] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4943.278064] Lustre: 206033:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4944.943301] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4947.188153] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4948.736461] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4951.820298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4951.907394] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4954.610702] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4956.878211] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4959.727457] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4959.767340] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4960.217531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4962.126221] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4964.474510] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4970.306620] Lustre: lustre-OST0000: Not available for connect from 192.168.203.2@tcp (stopping) [ 4970.314885] Lustre: Skipped 4 previous similar messages [ 4970.470466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4970.470679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4970.476696] Lustre: Skipped 1 previous similar message [ 4980.192638] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4980.216959] LustreError: 209238:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4980.220345] LustreError: 209238:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4980.248781] Lustre: server umount lustre-OST0000 complete [ 4980.538430] LustreError: 207663:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4980.548299] LustreError: 207663:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 4996.071256] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4997.896515] Lustre: server umount lustre-OST0001 complete [ 5006.031529] Lustre: server umount lustre-MDT0000 complete [ 5006.309451] LustreError: 206044:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5006.321216] LustreError: 206044:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 10 previous similar messages [ 5008.034551] LustreError: 206765:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289312 with bad export cookie 5111080691324143016 [ 5008.040068] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5008.209425] Lustre: server umount lustre-MDT0001 complete [ 5010.801842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5012.526608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5014.093912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5015.760077] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5018.565686] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:02:01 (1761289321) [ 5025.421388] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 5029.390412] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 5033.406303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5036.466856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5039.255746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5042.226475] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5045.360707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5045.399894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5045.509880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5045.532328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5045.577821] Lustre: lustre-MDT0000: new disk, initializing [ 5045.629102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5045.631837] Lustre: Skipped 3 previous similar messages [ 5045.648050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5047.565799] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5053.029780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5053.062719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5053.110748] Lustre: 213663:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5053.137699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5053.140295] Lustre: Skipped 1 previous similar message [ 5053.179449] Lustre: lustre-MDT0001: new disk, initializing [ 5053.242541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5053.248850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5053.279567] Lustre: 214463:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 5053.286564] Lustre: 214463:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5053.291040] Lustre: 214463:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 5053.296044] Lustre: 214463:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 5053.303302] Lustre: 214463:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5053.307986] Lustre: 214463:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5054.909851] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5058.762229] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5060.344378] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5063.245628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5063.278400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5063.385797] Lustre: lustre-OST0000: new disk, initializing [ 5063.388219] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5063.602237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5063.607067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5063.659773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5065.750368] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5069.370476] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5071.121993] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5071.218211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5072.957234] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5073.049148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5079.009307] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5079.010461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5079.012834] LustreError: Skipped 2 previous similar messages [ 5079.017636] Lustre: Skipped 4 previous similar messages [ 5079.019051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5079.023758] Lustre: Skipped 10 previous similar messages [ 5088.223189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5088.245203] LustreError: 216606:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5088.247262] LustreError: 216606:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5088.318092] Lustre: server umount lustre-OST0000 complete [ 5096.047450] Lustre: server umount lustre-MDT0000 complete [ 5097.566921] LustreError: 213654:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289401 with bad export cookie 5111080691324144465 [ 5097.570472] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5097.572355] LustreError: 213654:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5097.732161] Lustre: server umount lustre-MDT0001 complete [ 5101.000626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5101.191487] LustreError: 217485:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5102.771721] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5106.316726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5108.086919] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5110.489189] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5112.053527] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5114.845789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5114.993723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5114.997170] Lustre: Skipped 4 previous similar messages [ 5117.296931] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5119.495392] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5121.097746] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5126.372691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 5128.122619] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5128.216908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5131.747462] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5131.751544] LustreError: Skipped 2 previous similar messages [ 5135.184200] Lustre: server umount lustre-OST0000 complete [ 5137.816348] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5139.812407] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5164.218206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5166.751065] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5169.095319] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5169.768794] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5177.069163] LustreError: 221896:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5177.071301] LustreError: 221896:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 5177.137471] Lustre: server umount lustre-OST0000 complete [ 5182.433132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5182.438393] Lustre: Skipped 6 previous similar messages [ 5184.884158] Lustre: server umount lustre-MDT0000 complete [ 5186.746399] LustreError: 217465:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289490 with bad export cookie 5111080691324145375 [ 5186.748741] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5186.752480] LustreError: 217465:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5186.941044] Lustre: server umount lustre-MDT0001 complete [ 5190.876561] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:04:53 (1761289493) [ 5193.976704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5194.173864] LustreError: 222973:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5195.817238] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5199.444658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5201.239333] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5203.506390] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5205.067480] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5207.929792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5210.366297] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5212.571182] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5214.263172] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5219.488248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 5220.983301] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5221.072117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5223.847614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5223.893375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5223.965962] Lustre: lustre-OST0001: new disk, initializing [ 5223.968910] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5226.357719] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5229.936235] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5231.597945] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5231.634070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5231.638306] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5231.685562] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5232.734778] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5234.278258] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5234.369708] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5236.710156] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5236.713255] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5236.713375] LustreError: Skipped 1 previous similar message [ 5236.716028] Lustre: Skipped 12 previous similar messages [ 5237.188438] Lustre: server umount lustre-OST0001 complete [ 5239.524339] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5239.618960] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5241.256776] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5241.351921] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5265.452718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5265.548128] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5265.551258] Lustre: Skipped 5 previous similar messages [ 5267.750997] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5269.813056] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5270.437155] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5271.019099] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5271.024244] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5271.025222] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5271.031110] Lustre: Skipped 1 previous similar message [ 5273.210542] Lustre: server umount lustre-OST0001 complete [ 5291.633127] Lustre: server umount lustre-OST0000 complete [ 5299.380173] Lustre: server umount lustre-MDT0000 complete [ 5300.920613] LustreError: 226012:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289605 with bad export cookie 5111080691324146306 [ 5300.923388] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5300.926029] LustreError: 226012:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5307.079313] LustreError: 229497:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5307.081486] LustreError: 229497:0:(obd_class.h:479:obd_check_dev()) Skipped 37 previous similar messages [ 5307.184415] Lustre: server umount lustre-MDT0001 complete [ 5309.399581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5310.897548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5312.416084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5313.936543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5316.625986] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:06:59 (1761289619) [ 5319.238349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5319.360193] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5319.372253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5320.959847] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5324.387389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5324.438652] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5324.455865] Lustre: 230989:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5326.190997] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5328.596159] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5330.195894] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5333.099696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5333.185562] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5335.647272] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5337.939960] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5342.510139] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5342.551653] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5344.993257] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5347.202646] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5348.883378] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 5350.464630] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5350.559390] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5352.089120] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5352.188468] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5353.613707] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c0bc759a000.ost_server_uuid 50 [ 5354.225638] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9c0bc759a000.ost_server_uuid in FULL state after 0 sec [ 5354.621905] Lustre: Permanently deactivating lustre-OST0001 [ 5354.629547] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5355.682510] Lustre: Permanently reactivating lustre-OST0001 [ 5358.048975] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5358.054690] Lustre: Skipped 9 previous similar messages [ 5363.138504] Lustre: server umount lustre-OST0001 complete [ 5363.170434] LustreError: 232615:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5363.176591] LustreError: 232615:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 25 previous similar messages [ 5367.264202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5367.267957] LustreError: Skipped 7 previous similar messages [ 5370.667932] Lustre: server umount lustre-OST0000 complete [ 5378.277790] Lustre: server umount lustre-MDT0000 complete [ 5379.753235] LustreError: 232630:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289683 with bad export cookie 5111080691324147594 [ 5379.756605] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5379.759362] LustreError: 232630:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5379.925031] Lustre: server umount lustre-MDT0001 complete [ 5382.097095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5383.708852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5385.276269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5386.837542] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5389.416986] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:08:12 (1761289692) [ 5389.807557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5392.331824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5392.475746] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5392.492812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5392.497100] Lustre: Skipped 5 previous similar messages [ 5394.131252] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5397.648459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5397.718018] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5397.735050] Lustre: 237134:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5399.450374] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5401.787493] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5403.327875] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5406.189936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5406.222053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5406.315310] Lustre: Permanently deactivating lustre-OST0000 [ 5406.323243] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5408.709868] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5409.076305] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5410.873824] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5413.795064] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5416.210963] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5418.328762] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5419.040883] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5424.399566] Lustre: Permanently reactivating lustre-OST0000 [ 5424.406707] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5424.409048] Lustre: Skipped 2 previous similar messages [ 5429.247785] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5429.251094] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.256520] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5429.273335] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5429.281484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5435.598754] Lustre: server umount lustre-OST0001 complete [ 5443.357503] Lustre: server umount lustre-OST0000 complete [ 5453.047118] Lustre: server umount lustre-MDT0000 complete [ 5456.812484] LustreError: 237126:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289760 with bad export cookie 5111080691324149225 [ 5456.818739] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5456.829718] LustreError: 237126:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5457.358569] Lustre: server umount lustre-MDT0001 complete [ 5468.012720] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 5470.919907] Key type lgssc unregistered [ 5471.246523] LNet: 241527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5471.258556] LNetError: 241527:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5471.274515] LNet: Removed LNI 192.168.203.102@tcp [ 5472.408261] Key type .llcrypt unregistered [ 5472.409907] Key type ._llcrypt unregistered [ 5494.927605] Key type ._llcrypt registered [ 5494.929476] Key type .llcrypt registered [ 5494.993279] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 5495.759059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5495.821320] alg: No test for adler32 (adler32-zlib) [ 5496.803467] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 5496.952394] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 5498.578316] Key type lgssc registered [ 5499.258106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5501.340682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5505.342253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5505.357664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5506.675540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5509.085547] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5515.056678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5515.103860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5515.157489] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5515.184747] Lustre: 242965:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5515.191731] Lustre: Permanently deactivating lustre-MDT0001 [ 5515.203452] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5515.419215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5515.821792] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5515.837614] LustreError: 243006:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5515.847431] LustreError: 243006:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5515.856275] LustreError: 243006:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 5518.157305] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5521.865889] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5524.039960] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5528.084884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5528.293720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5531.365310] LustreError: 244598:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5531.372312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5531.376059] LustreError: 244598:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 5531.701264] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5534.676989] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5536.738710] LustreError: 244598:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5538.606952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5538.762447] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5540.790626] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5540.801136] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5542.713318] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5546.342608] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5548.542509] Lustre: Permanently reactivating lustre-MDT0001 [ 5548.549359] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5548.552411] Lustre: Skipped 2 previous similar messages [ 5556.082574] Lustre: lustre-MDT0001: Client 5aa33626-be3c-46a7-a23d-f7ab51a9afb0 (at 192.168.203.2@tcp) reconnecting [ 5558.329919] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5559.367567] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5562.690431] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5562.817549] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5565.276290] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5565.413857] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5566.207461] Lustre: 242972:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 5566.215602] Lustre: 242972:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5566.218777] Lustre: 242972:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 5566.225566] Lustre: 242972:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 5566.230630] Lustre: 242972:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5566.234444] Lustre: 242972:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5567.192254] Lustre: Permanently deactivating lustre-MDT0001 [ 5567.205526] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5567.208905] Lustre: Skipped 2 previous similar messages [ 5573.686572] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5575.445623] Lustre: 244374:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 506 < left 979, rollback = 2 [ 5575.450969] Lustre: 244374:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 5575.456364] Lustre: 244374:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 2/8/6, destroy: 0/0/0 [ 5575.459814] Lustre: 244374:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5575.463733] Lustre: 244374:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 6/979/0 [ 5575.467025] Lustre: 244374:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5575.471689] Lustre: 244374:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 5575.475953] Lustre: 244374:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5575.480199] Lustre: 244374:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 8/130/4, delete: 0/0/0 [ 5575.484364] Lustre: 244374:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5575.487952] Lustre: 244374:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0 [ 5575.491625] Lustre: 244374:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5581.794879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5581.797317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5581.802034] Lustre: Skipped 2 previous similar messages [ 5581.802591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5581.807955] Lustre: Skipped 2 previous similar messages [ 5583.690721] LustreError: 247042:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5583.750821] Lustre: server umount lustre-MDT0000 complete [ 5585.931582] LustreError: 242956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289890 with bad export cookie 10432744911354455673 [ 5585.933592] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5585.939745] LustreError: 242956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5586.028939] LustreError: 247243:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5586.031832] LustreError: 247243:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5586.177876] Lustre: server umount lustre-MDT0001 complete [ 5594.362436] LustreError: 247443:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5594.365093] LustreError: 247443:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5594.389313] Lustre: server umount lustre-OST0000 complete [ 5600.605420] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:11:43 (1761289903) [ 5605.989093] LustreError: 248448:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5605.991561] LustreError: 248448:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5606.081685] Lustre: server umount lustre-OST0001 complete [ 5609.408681] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 5615.928044] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 5622.796204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5627.155440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5631.060195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5635.277956] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5639.257178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5639.304814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5639.469104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5639.480602] Lustre: Skipped 2 previous similar messages [ 5639.510103] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5639.557416] Lustre: lustre-MDT0000: new disk, initializing [ 5639.618453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5639.645528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5642.503590] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5650.495884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5650.543103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5650.640821] Lustre: 250630:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5650.683100] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5650.687973] Lustre: Skipped 1 previous similar message [ 5650.747765] Lustre: lustre-MDT0001: new disk, initializing [ 5650.826601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5650.867441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5650.875064] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5650.958295] Lustre: 251427:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 5650.970922] Lustre: 251427:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 3 previous similar messages [ 5650.974553] Lustre: 251427:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5650.985625] Lustre: 251427:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 5650.989456] Lustre: 251427:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 5650.999223] Lustre: 251427:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 5651.010860] Lustre: 251427:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 5651.027838] Lustre: 251427:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 5651.042780] Lustre: 251427:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5651.046368] Lustre: 251427:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 5651.053815] Lustre: 251427:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5651.067506] Lustre: 251427:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 5653.206439] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5658.777930] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5661.348864] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5666.633291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5666.705778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5666.884197] Lustre: lustre-OST0000: new disk, initializing [ 5666.886940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5666.972777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5670.600463] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5673.021252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5673.025412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5673.140661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5675.780854] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5677.402217] LustreError: 250638:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5684.198753] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5684.250343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5684.349555] Lustre: lustre-OST0001: new disk, initializing [ 5684.358345] Lustre: lustre-OST0001: Not available for connect from 192.168.203.2@tcp (not set up) [ 5684.361944] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5687.473381] LustreError: 250638:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5687.491918] LustreError: 250638:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5688.290948] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5693.310935] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5694.447525] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5694.488319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5694.494906] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5694.582565] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5697.535792] LustreError: 250638:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5699.560524] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5699.567378] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5699.575096] Lustre: Skipped 1 previous similar message [ 5699.580927] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5699.584078] Lustre: Skipped 1 previous similar message [ 5700.665358] LustreError: 254125:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5700.670370] LustreError: 254125:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5700.720418] Lustre: server umount lustre-OST0001 complete [ 5704.675412] LustreError: 252361:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5704.689519] LustreError: 252361:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 5709.795534] LustreError: 252361:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5709.804651] LustreError: 252361:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 5714.918855] LustreError: 252363:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5714.927954] LustreError: 252363:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 5719.520505] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5719.520977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5719.528433] LustreError: Skipped 2 previous similar messages [ 5719.533733] Lustre: Skipped 1 previous similar message [ 5719.534575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5719.534579] Lustre: Skipped 1 previous similar message [ 5720.568119] LustreError: 254347:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5720.583112] LustreError: 254347:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5720.703556] Lustre: server umount lustre-OST0000 complete [ 5725.153453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5725.156503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5725.164869] Lustre: Skipped 2 previous similar messages [ 5725.176131] Lustre: Skipped 4 previous similar messages [ 5730.280158] LustreError: 250642:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5730.290525] LustreError: 250642:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 5731.587504] Lustre: server umount lustre-MDT0000 complete [ 5734.129331] LustreError: 252378:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290038 with bad export cookie 10432744911354458263 [ 5734.134866] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5734.144026] LustreError: 252378:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5740.341445] LustreError: 254751:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5740.346400] LustreError: 254751:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5740.498920] Lustre: server umount lustre-MDT0001 complete [ 5747.537344] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 5749.779369] Key type lgssc unregistered [ 5750.022635] LNet: 255284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5750.034170] LNetError: 255284:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5750.046120] LNet: Removed LNI 192.168.203.102@tcp [ 5750.625332] Key type .llcrypt unregistered [ 5750.632942] Key type ._llcrypt unregistered [ 5772.699230] Key type ._llcrypt registered [ 5772.700757] Key type .llcrypt registered [ 5772.773215] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 5781.620876] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 5782.140593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5782.154863] alg: No test for adler32 (adler32-zlib) [ 5783.060272] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 5783.177536] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 5784.791499] Key type lgssc registered [ 5785.358358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5790.688511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5794.724187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5798.560231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5802.832573] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5807.292361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5807.319745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5807.332205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5808.480407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5808.508944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5808.557961] Lustre: lustre-MDT0000: new disk, initializing [ 5808.617642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5808.633226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5810.961519] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5816.886079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5816.923569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5816.974752] Lustre: 259212:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5817.009710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5817.012938] Lustre: Skipped 1 previous similar message [ 5817.054588] Lustre: lustre-MDT0001: new disk, initializing [ 5817.102665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5817.122817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5817.131420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5817.174123] Lustre: 260011:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 5817.181672] Lustre: 260011:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5817.191017] Lustre: 260011:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 5817.196220] Lustre: 260011:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 5817.200713] Lustre: 260011:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5817.208542] Lustre: 260011:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5819.079902] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5823.344938] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5825.129385] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5828.582962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5828.622073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5828.753643] Lustre: lustre-OST0000: new disk, initializing [ 5828.758158] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5828.811242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5831.886415] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5835.873287] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5837.628211] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5837.829303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5837.833889] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5837.885159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5838.787230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5840.583044] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5840.677686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5842.916994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5842.917155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5842.925428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5842.929186] LustreError: Skipped 1 previous similar message [ 5847.669517] LustreError: 262160:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5847.757430] Lustre: server umount lustre-OST0000 complete [ 5853.154353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5853.155502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5853.155614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5853.164615] Lustre: Skipped 2 previous similar messages [ 5853.173768] Lustre: Skipped 2 previous similar messages [ 5855.768408] LustreError: 262361:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5855.771990] LustreError: 262361:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5855.839760] Lustre: server umount lustre-MDT0000 complete [ 5858.131431] LustreError: 259204:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290162 with bad export cookie 1602337704545148816 [ 5858.136806] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5858.139049] LustreError: 259204:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5858.283486] LustreError: 262562:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5858.285571] LustreError: 262562:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5858.458864] Lustre: server umount lustre-MDT0001 complete [ 5864.161364] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 03:16:06 (1761290166) [ 5868.180474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5868.479399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5870.686760] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5873.635058] LustreError: 263231:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5873.644387] LustreError: 263231:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 5875.308073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5875.543794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5877.875211] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5880.884094] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5882.631125] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5886.247487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5886.420206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5889.416486] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5891.937340] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5944.800946] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5944.807634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5944.815763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5946.848031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5946.852311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5946.853517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5946.861437] Lustre: Skipped 1 previous similar message [ 5948.276929] LustreError: 265555:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5948.280094] LustreError: 265555:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5948.380726] Lustre: server umount lustre-OST0000 complete [ 5950.758746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5958.066155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5958.269642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5960.631072] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 5963.332296] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5967.462197] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.467706] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5972.581034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.587299] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5998.561716] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5998.562297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5998.564918] LustreError: Skipped 1 previous similar message [ 5998.571074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5998.574375] Lustre: Skipped 1 previous similar message [ 6003.822095] LustreError: 267217:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6003.824756] LustreError: 267217:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6003.924281] Lustre: server umount lustre-OST0000 complete [ 6008.801050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6008.801357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6008.802128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6008.802146] Lustre: Skipped 2 previous similar messages [ 6008.806350] Lustre: Skipped 2 previous similar messages [ 6011.495708] Lustre: server umount lustre-MDT0000 complete [ 6012.905417] LustreError: 263212:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290317 with bad export cookie 1602337704545149726 [ 6012.908580] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6012.910360] LustreError: 263212:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6012.979373] LustreError: 267622:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6012.982253] LustreError: 267622:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6013.072722] Lustre: server umount lustre-MDT0001 complete [ 6017.265098] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 6018.468641] Key type lgssc unregistered [ 6018.610403] LNet: 268153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6018.613708] LNetError: 268153:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6018.624297] LNet: Removed LNI 192.168.203.102@tcp [ 6018.964905] Key type .llcrypt unregistered [ 6018.966244] Key type ._llcrypt unregistered [ 6028.923845] Key type ._llcrypt registered [ 6028.925372] Key type .llcrypt registered [ 6028.967570] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 6029.341788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6029.381532] alg: No test for adler32 (adler32-zlib) [ 6030.240245] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 6030.331429] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 6031.919179] Key type lgssc registered [ 6032.330250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6035.322682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6035.333384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6036.536290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6038.094135] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6041.440510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6041.573951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6042.967342] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6045.026240] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6046.449448] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6049.029196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6049.167894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6051.469668] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6053.552691] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6061.479821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6076.898157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6076.898947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.902047] LustreError: Skipped 1 previous similar message [ 6076.907777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6076.912181] Lustre: Skipped 1 previous similar message [ 6079.021759] LustreError: 272570:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6079.103365] Lustre: server umount lustre-OST0000 complete [ 6082.017081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6082.017797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6082.023459] Lustre: Skipped 2 previous similar messages [ 6082.027669] Lustre: Skipped 1 previous similar message [ 6086.722265] LustreError: 272772:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6086.725112] LustreError: 272772:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6086.757736] Lustre: server umount lustre-MDT0000 complete [ 6087.136037] LustreError: 269527:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6087.142733] LustreError: 269527:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 6088.215385] LustreError: 271147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290392 with bad export cookie 12582803310689713294 [ 6088.218201] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6088.221202] LustreError: 271147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6088.291067] LustreError: 272975:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6088.293220] LustreError: 272975:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6088.391253] Lustre: server umount lustre-MDT0001 complete [ 6092.756872] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 6094.043611] Key type lgssc unregistered [ 6094.189669] LNet: 273506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6094.193391] LNetError: 273506:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6094.203429] LNet: Removed LNI 192.168.203.102@tcp [ 6094.582401] Key type .llcrypt unregistered [ 6094.584079] Key type ._llcrypt unregistered [ 6103.032265] Key type ._llcrypt registered [ 6103.033813] Key type .llcrypt registered [ 6103.083487] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 6103.448428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6103.458941] alg: No test for adler32 (adler32-zlib) [ 6104.347896] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 6104.448077] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 6106.039186] Key type lgssc registered [ 6106.497829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6109.984705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6109.993624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6111.219521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6112.814135] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6116.652709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6116.851378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6118.797705] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6121.236350] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6122.931739] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6125.838402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6126.002460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6128.324268] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6130.633891] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6136.239981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6146.529800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6146.530026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6146.531234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6146.541822] Lustre: Skipped 1 previous similar message [ 6151.649429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6151.652443] Lustre: Skipped 2 previous similar messages [ 6152.242351] LustreError: 277101:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6152.314936] Lustre: server umount lustre-OST0000 complete [ 6156.769980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6156.770825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6156.777260] Lustre: Skipped 1 previous similar message [ 6160.069306] LustreError: 277302:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6160.072135] LustreError: 277302:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6160.111856] Lustre: server umount lustre-MDT0000 complete [ 6161.889383] LustreError: 274681:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6161.897013] LustreError: 274681:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 6161.906604] LustreError: 274666:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290466 with bad export cookie 2956650103737566678 [ 6161.909874] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6161.915425] LustreError: 274666:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6162.003862] LustreError: 277503:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6162.009062] LustreError: 277503:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6162.135158] Lustre: server umount lustre-MDT0001 complete [ 6166.834415] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 6168.034873] Key type lgssc unregistered [ 6168.195869] LNet: 278035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6168.199754] LNetError: 278035:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6168.207776] LNet: Removed LNI 192.168.203.102@tcp [ 6168.582469] Key type .llcrypt unregistered [ 6168.584257] Key type ._llcrypt unregistered [ 6179.583048] Key type ._llcrypt registered [ 6179.584666] Key type .llcrypt registered [ 6179.646231] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 6180.110599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6180.130646] alg: No test for adler32 (adler32-zlib) [ 6181.023806] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 6181.133723] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 6182.743324] Key type lgssc registered [ 6183.261979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6186.633044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6186.642169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6187.855271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6189.554506] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6193.439906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6193.601907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6195.278902] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6197.643829] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6199.233950] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6202.290423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6202.450269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6204.891125] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6207.567478] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6212.715220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6223.329941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6223.330110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6223.335508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6223.340287] Lustre: Skipped 1 previous similar message [ 6228.451309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6228.454140] Lustre: Skipped 1 previous similar message [ 6229.166145] LustreError: 282428:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6229.242911] Lustre: server umount lustre-OST0000 complete [ 6233.569702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6233.571847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6233.575948] Lustre: Skipped 2 previous similar messages [ 6238.688863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6238.691320] Lustre: Skipped 1 previous similar message [ 6243.809280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6243.811722] Lustre: Skipped 1 previous similar message [ 6245.343202] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6245.392563] LustreError: 282629:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6245.395611] LustreError: 282629:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6245.437758] Lustre: server umount lustre-MDT0000 complete [ 6246.998510] LustreError: 279390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290551 with bad export cookie 12093158441096058538 [ 6247.000305] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6247.003129] LustreError: 279390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6247.076583] LustreError: 282831:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6247.079563] LustreError: 282831:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6247.172378] Lustre: server umount lustre-MDT0001 complete [ 6251.338668] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 6252.503203] Key type lgssc unregistered [ 6252.661520] LNet: 283362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6252.664721] LNetError: 283362:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6252.673524] LNet: Removed LNI 192.168.203.102@tcp [ 6253.042569] Key type .llcrypt unregistered [ 6253.043991] Key type ._llcrypt unregistered [ 6261.203641] Key type ._llcrypt registered [ 6261.205133] Key type .llcrypt registered [ 6261.257282] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 6261.721879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6261.764233] alg: No test for adler32 (adler32-zlib) [ 6262.639034] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 6262.742444] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 6264.335200] Key type lgssc registered [ 6264.835865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6268.328864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6268.341761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6269.620890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6271.846281] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6276.889711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6277.122384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6278.980429] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6282.132881] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6283.936773] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6287.300765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6287.458719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6289.815810] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6292.078471] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6294.629674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6308.321333] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6308.321489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6308.322487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6308.332613] Lustre: Skipped 1 previous similar message [ 6312.501145] LustreError: 286993:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6312.585618] Lustre: server umount lustre-OST0000 complete [ 6318.048206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6318.051858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6318.059143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6318.061444] Lustre: Skipped 1 previous similar message [ 6320.511673] LustreError: 287195:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6320.513852] LustreError: 287195:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6320.548120] Lustre: server umount lustre-MDT0000 complete [ 6322.081495] LustreError: 284522:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290626 with bad export cookie 10342260799015750074 [ 6322.085137] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6322.086623] LustreError: 284522:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6322.154720] LustreError: 287397:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6322.156814] LustreError: 287397:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6322.253411] Lustre: server umount lustre-MDT0001 complete [ 6326.335451] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 6327.504403] Key type lgssc unregistered [ 6327.657908] LNet: 287927:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6327.662166] LNetError: 287927:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6327.674322] LNet: Removed LNI 192.168.203.102@tcp [ 6328.024755] Key type .llcrypt unregistered [ 6328.026479] Key type ._llcrypt unregistered [ 6332.447173] Key type ._llcrypt registered [ 6332.448762] Key type .llcrypt registered [ 6332.504486] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6339.517904] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 6346.463284] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 6346.848737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6346.895347] alg: No test for adler32 (adler32-zlib) [ 6347.774108] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 6347.870584] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 6349.463152] Key type lgssc registered [ 6349.885535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6353.349935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6356.501369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6359.443738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6362.406488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6365.689038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6365.713054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6365.720637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.840852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6366.861159] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6366.930650] Lustre: lustre-MDT0000: new disk, initializing [ 6366.985114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6367.001754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6368.797369] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6373.971779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6374.000506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6374.044207] Lustre: 291408:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6374.075746] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6374.078357] Lustre: Skipped 1 previous similar message [ 6374.119830] Lustre: lustre-MDT0001: new disk, initializing [ 6374.166167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6374.184794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6374.189932] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6374.225148] Lustre: 292209:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6374.232901] Lustre: 292209:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6374.236094] Lustre: 292209:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6374.240169] Lustre: 292209:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6374.245780] Lustre: 292209:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6374.252935] Lustre: 292209:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6375.905808] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6379.683625] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6381.425494] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6384.485124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6384.526085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6384.653733] Lustre: lustre-OST0000: new disk, initializing [ 6384.657428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6384.703965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6387.387501] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6390.285113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6390.289717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6390.337047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6391.501302] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6393.226619] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6393.328729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6394.970525] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6395.065104] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6400.483644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6400.483658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6400.490130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6400.495176] Lustre: Skipped 1 previous similar message [ 6405.602637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6405.605199] Lustre: Skipped 1 previous similar message [ 6410.207136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6410.228863] LustreError: 294350:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6410.309764] Lustre: server umount lustre-OST0000 complete [ 6415.328041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6415.331520] LustreError: Skipped 1 previous similar message [ 6415.334020] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6415.339312] Lustre: Skipped 1 previous similar message [ 6415.343302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6418.187591] LustreError: 294552:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6418.190330] LustreError: 294552:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6418.226611] Lustre: server umount lustre-MDT0000 complete [ 6419.803766] LustreError: 291401:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290723 with bad export cookie 8610554736084505359 [ 6419.806372] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6419.808321] LustreError: 291401:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6419.883731] LustreError: 294755:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6419.885892] LustreError: 294755:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6419.990148] Lustre: server umount lustre-MDT0001 complete [ 6423.611380] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:25:26 (1761290726) [ 6426.400170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6426.668863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6428.343293] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6431.713567] LustreError: 295425:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6431.724139] LustreError: 295425:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 6432.093776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6432.297689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6434.005605] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6436.284271] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6437.883910] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6441.036988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6441.217826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6443.683125] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6446.016729] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6456.803068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6456.806651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6456.808676] Lustre: Skipped 1 previous similar message [ 6456.813037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6456.816619] Lustre: Skipped 1 previous similar message [ 6460.983142] LustreError: 297676:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6460.988090] LustreError: 297676:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6461.073706] Lustre: server umount lustre-OST0000 complete [ 6462.945116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6462.949459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6462.956049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6467.041974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6472.160097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6472.162413] Lustre: Skipped 3 previous similar messages [ 6477.279137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6477.321481] LustreError: 297877:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6477.324389] LustreError: 297877:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6477.355462] Lustre: server umount lustre-MDT0000 complete [ 6478.916229] LustreError: 295405:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290783 with bad export cookie 8610554736084506269 [ 6478.918224] LustreError: 296147:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 6478.920351] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6478.922865] LustreError: 295405:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6479.121242] Lustre: server umount lustre-MDT0001 complete [ 6483.513844] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 6484.762189] Key type lgssc unregistered [ 6484.911495] LNet: 298612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6484.915641] LNetError: 298612:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6484.924655] LNet: Removed LNI 192.168.203.102@tcp [ 6485.308188] Key type .llcrypt unregistered [ 6485.309523] Key type ._llcrypt unregistered [ 6495.915979] Key type ._llcrypt registered [ 6495.917573] Key type .llcrypt registered [ 6495.965925] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 6502.771814] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 6503.244948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6503.290812] alg: No test for adler32 (adler32-zlib) [ 6504.166400] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 6504.273210] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 6505.871418] Key type lgssc registered [ 6506.325964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6509.720943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6512.372249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6515.020393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6517.700184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6519.836257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6522.005411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6524.998310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6525.027108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6525.036763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6526.145530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6526.164357] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6526.205432] Lustre: lustre-MDT0000: new disk, initializing [ 6526.239770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6526.248756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6527.843536] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6533.023063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6533.052673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6533.102096] Lustre: 302388:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6533.134437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6533.137384] Lustre: Skipped 1 previous similar message [ 6533.181896] Lustre: lustre-MDT0001: new disk, initializing [ 6533.235417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6533.253324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6533.258729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6533.298842] Lustre: 303187:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6533.303517] Lustre: 303187:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6533.306180] Lustre: 303187:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6533.314964] Lustre: 303187:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6533.320307] Lustre: 303187:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6533.324504] Lustre: 303187:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6534.844258] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6538.266022] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6539.669653] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6542.215954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6542.245221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6542.359439] Lustre: lustre-OST2710: new disk, initializing [ 6542.361742] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6542.397313] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6544.698163] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6548.134913] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 6548.992883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6548.997677] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6549.044398] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6550.826635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6550.852816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6550.921143] Lustre: lustre-OST03e8: new disk, initializing [ 6550.924098] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6550.963061] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6553.159451] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6555.461550] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 6559.246103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6559.250811] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6559.303095] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6559.666304] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6559.765887] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6561.439619] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6561.545893] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6563.227422] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6563.334606] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6565.052256] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6565.150106] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6568.927921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6568.931893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6568.938117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6569.445636] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6569.445947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6569.453910] Lustre: Skipped 2 previous similar messages [ 6572.485957] LustreError: 306732:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6572.522665] Lustre: server umount lustre-MDT0000 complete [ 6573.962454] LustreError: 302380:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290878 with bad export cookie 2253836579243777654 [ 6573.966131] LustreError: 302380:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6573.966330] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6574.041326] LustreError: 306934:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6574.044018] LustreError: 306934:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6574.138735] Lustre: server umount lustre-MDT0001 complete [ 6585.849164] LustreError: 307134:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6585.851695] LustreError: 307134:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6585.882530] Lustre: server umount lustre-OST2710 complete [ 6597.683349] LustreError: 307336:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6597.686078] LustreError: 307336:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6597.779661] Lustre: server umount lustre-OST03e8 complete [ 6605.576372] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 6609.162367] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 6612.930702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6615.419356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6617.928029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6620.513544] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6623.430665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6623.458364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6623.580731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6623.597285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6623.638693] Lustre: lustre-MDT0000: new disk, initializing [ 6623.684601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6623.695948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6625.446509] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6630.625950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6630.669881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6630.714505] Lustre: 310129:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6630.869139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6630.940882] Lustre: 310930:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6630.947360] Lustre: 310930:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 6630.951831] Lustre: 310930:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6630.956233] Lustre: 310930:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6630.963393] Lustre: 310930:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6630.971330] Lustre: 310930:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6630.977974] Lustre: 310930:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6630.982405] Lustre: 310930:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6630.986555] Lustre: 310930:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6630.990996] Lustre: 310930:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6630.994571] Lustre: 310930:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6630.998711] Lustre: 310930:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6633.500286] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6641.391063] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6644.997655] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6651.374640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6651.442919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6651.609868] Lustre: lustre-OST0000: new disk, initializing [ 6651.612847] Lustre: Skipped 1 previous similar message [ 6651.615934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6651.618166] Lustre: Skipped 2 previous similar messages [ 6651.701755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6651.716354] Lustre: Skipped 1 previous similar message [ 6656.809575] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6658.114970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6658.124083] Lustre: Skipped 1 previous similar message [ 6658.135254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6658.273758] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6664.807756] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6669.887703] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6670.231819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6674.586378] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6674.850501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6679.008182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6679.009135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6679.035887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6679.071682] Lustre: Skipped 3 previous similar messages [ 6679.128139] Lustre: Skipped 1 previous similar message [ 6684.142593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6684.145297] Lustre: Skipped 1 previous similar message [ 6689.259741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6689.274827] Lustre: Skipped 1 previous similar message [ 6691.295166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6691.352803] LustreError: 313075:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6691.355572] LustreError: 313075:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6691.750952] Lustre: server umount lustre-OST0000 complete [ 6697.444156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6697.452049] LustreError: Skipped 1 previous similar message [ 6697.460117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6697.487629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6702.218564] LustreError: 313278:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6702.221258] LustreError: 313278:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6702.456996] Lustre: server umount lustre-MDT0000 complete [ 6707.014692] LustreError: 310913:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761291011 with bad export cookie 2253836579243779369 [ 6707.015947] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6707.049884] LustreError: 310913:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6707.581263] Lustre: server umount lustre-MDT0001 complete [ 6716.846186] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:30:18 (1761291018) [ 6718.720362] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6720.766125] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:30:22 (1761291022) [ 6724.375723] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 6732.006714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6735.591186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6738.877461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6742.034077] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6743.097380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6749.582455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6749.785351] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6749.798760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6749.903688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6753.068400] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6760.473014] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6760.538675] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6760.563055] Lustre: 315611:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6764.760215] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6770.277910] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6774.624066] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6780.994916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6781.058573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6781.286201] Lustre: Denying initial registration attempt from nid 192.168.203.102@tcp, specified as failover [ 6781.307145] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6781.324289] LustreError: 317232:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -99 [ 6781.336442] LustreError: 317232:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 6781.354761] LustreError: 317232:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6781.526206] Lustre: server umount lustre-OST0000 complete [ 6781.528982] LustreError: 317232:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6787.042050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6787.045155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6787.054040] Lustre: Skipped 2 previous similar messages [ 6787.057314] Lustre: Skipped 2 previous similar messages [ 6801.375493] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6801.446339] LustreError: 317601:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6801.464565] LustreError: 317601:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6801.554402] Lustre: server umount lustre-MDT0000 complete [ 6802.400295] LustreError: 315622:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6802.417207] LustreError: 315622:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 4 previous similar messages [ 6804.246233] LustreError: 315603:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761291108 with bad export cookie 2253836579243780293 [ 6804.249035] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6804.272166] LustreError: 315603:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6804.538707] Lustre: server umount lustre-MDT0001 complete [ 6813.402946] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 6815.605901] Key type lgssc unregistered [ 6815.861556] LNet: 318336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6815.867535] LNetError: 318336:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6815.882799] LNet: Removed LNI 192.168.203.102@tcp [ 6816.496116] Key type .llcrypt unregistered [ 6816.498272] Key type ._llcrypt unregistered [ 6824.776165] Key type ._llcrypt registered [ 6824.777673] Key type .llcrypt registered [ 6824.871082] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 6825.669435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6825.705186] alg: No test for adler32 (adler32-zlib) [ 6826.692488] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 6826.909699] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 6828.591173] Key type lgssc registered [ 6829.533220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6835.142651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6837.999607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6841.168736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6844.276297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6845.121457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6859.095058] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 6865.870258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6865.886299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6867.098453] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6867.113781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6867.267817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6870.486353] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6878.446714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6878.516620] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6878.534268] Lustre: 321091:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6878.739995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6881.658606] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6885.946327] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6889.310794] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6896.893849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6897.001817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6897.167198] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6897.355856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6903.443348] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6909.191625] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6915.043376] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6915.052062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6915.063240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6915.553508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6915.557843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6915.565091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6917.575930] LustreError: 323379:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6917.740600] Lustre: server umount lustre-OST0000 complete [ 6924.771256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6924.787710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6924.807949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6927.122855] LustreError: 323581:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6927.127401] LustreError: 323581:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6927.259304] Lustre: server umount lustre-MDT0000 complete [ 6930.646252] LustreError: 321084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761291234 with bad export cookie 8048291676563056497 [ 6930.649787] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6930.656795] LustreError: 321084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6930.800944] LustreError: 323785:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6930.809018] LustreError: 323785:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6931.071742] Lustre: server umount lustre-MDT0001 complete [ 6941.483494] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 6944.438479] Key type lgssc unregistered [ 6944.783971] LNet: 324320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6944.796217] LNetError: 324320:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6945.832464] LNet: Removed LNI 192.168.203.102@tcp [ 6946.502874] Key type .llcrypt unregistered [ 6946.505617] Key type ._llcrypt unregistered [ 6968.293235] Key type ._llcrypt registered [ 6968.306327] Key type .llcrypt registered [ 6968.422551] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 6969.601111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6969.723814] alg: No test for adler32 (adler32-zlib) [ 6970.851876] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 6971.032188] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 6972.679157] Key type lgssc registered [ 6973.763324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6980.925523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6980.954137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6982.490752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6985.740564] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6992.872091] LustreError: 325727:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6992.893140] LustreError: 325727:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 6993.648380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6994.032133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6997.653716] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7002.562827] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7005.643276] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7011.469673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7011.831845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7016.818640] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7021.380548] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7038.434880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7038.436628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7038.471371] Lustre: Skipped 2 previous similar messages [ 7038.495379] Lustre: Skipped 2 previous similar messages [ 7042.391547] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.2@tcp (stopping) [ 7043.557040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7044.275168] LustreError: 327997:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7044.356747] Lustre: server umount lustre-MDT0000 complete [ 7047.493208] LustreError: 325722:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7047.518320] LustreError: 327349:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761291351 with bad export cookie 16457363868602351622 [ 7047.522491] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7047.530708] LustreError: 327349:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7048.678594] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7048.683232] LustreError: 327887:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7048.703031] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7048.703040] Lustre: Skipped 2 previous similar messages [ 7052.606928] LustreError: 325722:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7052.705507] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7052.710903] Lustre: Skipped 1 previous similar message [ 7053.723877] LustreError: 328200:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7053.728201] LustreError: 328200:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7053.910118] Lustre: server umount lustre-MDT0001 complete [ 7059.485429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7065.642784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7072.228188] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe4644d3e5f645df0 [ 7072.253885] Lustre: MGC192.168.203.102@tcp: Connection restored to 0@lo (at 0@lo) [ 7072.494430] LustreError: 327337:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7072.507291] LustreError: 327337:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 7072.566566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7072.602130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 7075.264526] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7081.454512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7081.815452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7082.861212] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7085.561340] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7090.261866] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7092.303430] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7095.726078] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7097.407835] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7102.435447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7102.444094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7102.460911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7103.464455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7107.327140] LustreError: 330505:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7107.338588] LustreError: 330505:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7107.489514] Lustre: server umount lustre-OST0000 complete [ 7112.672498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7112.677624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7112.689776] Lustre: Skipped 1 previous similar message [ 7116.241423] LustreError: 330707:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7116.245342] LustreError: 330707:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7116.323777] Lustre: server umount lustre-MDT0000 complete [ 7118.819678] LustreError: 329011:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7118.838283] LustreError: 329011:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 7120.238715] LustreError: 325708:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761291424 with bad export cookie 16457363868602359280 [ 7120.241501] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7120.269360] LustreError: 325708:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7120.470316] LustreError: 330909:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7120.482611] LustreError: 330909:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7120.714212] Lustre: server umount lustre-MDT0001 complete [ 7132.736841] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 7136.555982] Key type lgssc unregistered [ 7136.827634] LNet: 331441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7136.840213] LNetError: 331441:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7136.889395] LNet: Removed LNI 192.168.203.102@tcp [ 7137.711460] Key type .llcrypt unregistered [ 7137.718253] Key type ._llcrypt unregistered [ 7162.981867] Key type ._llcrypt registered [ 7162.983417] Key type .llcrypt registered [ 7163.137390] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 7164.936129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7165.071994] alg: No test for adler32 (adler32-zlib) [ 7166.344405] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 7166.825085] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 7168.639149] Key type lgssc registered [ 7170.451524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7181.479467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7181.532072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7183.691284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7188.961215] LustreError: 332839:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7188.978184] LustreError: 332839:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 7190.035499] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7193.121830] LustreError: 332839:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7198.181970] LustreError: 332839:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7202.633244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7202.929150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7209.875780] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7218.048876] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7222.762992] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7227.235975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7227.275410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7227.288702] Lustre: Skipped 1 previous similar message [ 7232.475797] LustreError: 334234:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7232.530954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7232.674278] Lustre: server umount lustre-MDT0000 complete [ 7235.969060] LustreError: 332819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761291540 with bad export cookie 5198045444171200597 [ 7235.989096] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7236.170921] LustreError: 334435:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7236.184498] LustreError: 334435:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7236.417941] Lustre: server umount lustre-MDT0001 complete [ 7249.641127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7249.866168] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7249.914154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7250.047485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7253.344649] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7260.330477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7260.406377] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7260.434581] Lustre: 335205:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7260.646390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7264.539988] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7268.782574] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7271.661407] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7278.354588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7278.557129] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7278.575991] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7278.587434] LustreError: 336868:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -2 [ 7278.590794] LustreError: 336868:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 7278.602357] LustreError: 336868:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 7278.726896] Lustre: server umount lustre-OST0000 complete [ 7278.730623] LustreError: 336868:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7286.169227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7286.279165] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7286.446159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7290.745245] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7292.529363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 7295.203948] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7300.747699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7300.816236] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7300.865164] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7300.945240] Lustre: lustre-OST0001: new disk, initializing [ 7300.948472] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7301.031951] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7302.385148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7302.389070] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7302.482058] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7305.647312] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7310.165070] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7312.864546] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7312.865249] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7312.867722] LustreError: Skipped 1 previous similar message [ 7312.871193] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7312.882134] Lustre: Skipped 2 previous similar messages [ 7312.892681] Lustre: Skipped 2 previous similar messages [ 7318.009858] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7318.784725] LustreError: 338993:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7318.787318] LustreError: 338993:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7318.850064] Lustre: server umount lustre-OST0001 complete [ 7323.106903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7323.109619] LustreError: 337336:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7323.110985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7323.133870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7323.136317] LustreError: 337336:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 7323.158184] Lustre: Skipped 2 previous similar messages [ 7328.226808] LustreError: 337338:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7328.245636] LustreError: 337338:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 7328.586124] LustreError: 339195:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7328.592331] LustreError: 339195:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7328.800441] Lustre: server umount lustre-OST0000 complete [ 7332.323063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7332.328571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7332.336363] Lustre: Skipped 1 previous similar message [ 7332.345067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7332.356733] Lustre: Skipped 2 previous similar messages [ 7338.156269] LustreError: 339397:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7338.160117] LustreError: 339397:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7338.266186] Lustre: server umount lustre-MDT0000 complete [ 7338.465858] LustreError: 336213:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7338.474650] LustreError: 336213:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 7341.117853] LustreError: 335197:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761291645 with bad export cookie 5198045444171201150 [ 7341.119729] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7341.133962] LustreError: 335197:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7341.447308] Lustre: server umount lustre-MDT0001 complete [ 7348.325115] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 7350.782987] Key type lgssc unregistered [ 7351.023676] LNet: 340130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7351.034296] LNetError: 340130:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7351.067193] LNet: Removed LNI 192.168.203.102@tcp [ 7351.837854] Key type .llcrypt unregistered [ 7351.839527] Key type ._llcrypt unregistered [ 7379.890673] Key type ._llcrypt registered [ 7379.892600] Key type .llcrypt registered [ 7379.981527] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 7391.483378] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 7392.152131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7392.200626] alg: No test for adler32 (adler32-zlib) [ 7393.198638] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 7393.420364] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 7395.055229] Key type lgssc registered [ 7396.122984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7403.964755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7410.161382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7416.592799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7423.588826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7433.401293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7433.457564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7433.485291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7435.001184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7435.062841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7435.195525] Lustre: lustre-MDT0000: new disk, initializing [ 7435.353723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7435.382607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7439.862749] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7450.768180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7450.843381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7450.905773] Lustre: 344062:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7450.953147] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7450.956311] Lustre: Skipped 1 previous similar message [ 7451.017856] Lustre: lustre-MDT0001: new disk, initializing [ 7451.110593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7451.146377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7451.161453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7451.226209] Lustre: 344858:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 7451.237166] Lustre: 344858:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 7451.256526] Lustre: 344858:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 7451.273839] Lustre: 344858:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 7451.276505] Lustre: 344858:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 7451.289208] Lustre: 344858:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 7455.070487] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7461.367563] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7464.661479] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7471.113521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7471.151858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7471.360439] Lustre: lustre-OST0000: new disk, initializing [ 7471.362904] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7471.464817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7476.962268] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7479.967703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7479.979371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7480.144559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7484.134041] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7488.216195] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7488.406696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7492.038314] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7492.258503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7495.651602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7495.652921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7495.670915] Lustre: Skipped 1 previous similar message [ 7500.095808] LustreError: 347002:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7500.324463] Lustre: server umount lustre-OST0000 complete [ 7505.892825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7505.897110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7505.908240] Lustre: Skipped 1 previous similar message [ 7505.911415] Lustre: Skipped 1 previous similar message [ 7510.300914] LustreError: 347205:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7510.307974] LustreError: 347205:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7510.599713] Lustre: server umount lustre-MDT0000 complete [ 7514.257746] LustreError: 345809:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761291818 with bad export cookie 778552138755321217 [ 7514.270909] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7514.281247] LustreError: 345809:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7514.441530] LustreError: 347406:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7514.461630] LustreError: 347406:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7515.013416] Lustre: server umount lustre-MDT0001 complete [ 7523.889201] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 03:43:45 (1761291825) [ 7530.039119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7535.805648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7555.941923] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 7564.445519] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 7572.502432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7578.693092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7583.514232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7588.212497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7593.534949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7593.608751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7593.806612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7593.843841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7593.938551] Lustre: lustre-MDT0000: new disk, initializing [ 7594.013757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7594.039657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7597.263602] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7607.520567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7607.601386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7607.742892] Lustre: 351184:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7607.848042] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7607.852076] Lustre: Skipped 1 previous similar message [ 7607.956795] Lustre: lustre-MDT0001: new disk, initializing [ 7608.054987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7608.104987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7608.111937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7608.218715] Lustre: 351981:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 7608.239586] Lustre: 351981:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 7608.252187] Lustre: 351981:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 7608.259174] Lustre: 351981:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7608.267225] Lustre: 351981:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 7608.274961] Lustre: 351981:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7608.287844] Lustre: 351981:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 7608.307395] Lustre: 351981:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7608.326823] Lustre: 351981:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 7608.347801] Lustre: 351981:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7608.361284] Lustre: 351981:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 7608.369162] Lustre: 351981:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7612.239205] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7620.223274] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7623.753817] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7630.721926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7630.836218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7631.100621] Lustre: lustre-OST0000: new disk, initializing [ 7631.113333] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7631.209171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7636.899634] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7640.105949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7640.115634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7640.189841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7643.733435] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7648.153613] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7648.548145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7653.237984] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7653.476915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7655.905347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7655.910672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7655.915504] LustreError: Skipped 1 previous similar message [ 7655.927923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7655.938536] Lustre: Skipped 2 previous similar messages [ 7660.513269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7660.521049] Lustre: Skipped 1 previous similar message [ 7661.625870] LustreError: 354125:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7661.634054] LustreError: 354125:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7661.942174] Lustre: server umount lustre-OST0000 complete [ 7665.633363] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7665.636172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7665.646213] Lustre: Skipped 2 previous similar messages [ 7671.610253] LustreError: 354326:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7671.615028] LustreError: 354326:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7671.883415] Lustre: server umount lustre-MDT0000 complete [ 7674.955595] LustreError: 351176:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761291979 with bad export cookie 778552138755322148 [ 7674.959794] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7674.960721] LustreError: 351176:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7675.252667] Lustre: server umount lustre-MDT0001 complete [ 7682.092972] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:46:24 (1761291984) [ 7688.951388] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:46:30 (1761291990) [ 7697.795409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7698.324840] LustreError: 355543:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7698.342180] LustreError: 355543:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 7698.385433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7702.049550] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7703.520627] LustreError: 355543:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7708.651142] LustreError: 355543:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7709.363602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7713.280167] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7718.627415] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7722.165494] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7728.441664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7733.547889] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7738.508606] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7752.153823] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7754.395540] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7756.343610] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7758.367324] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7760.435204] Lustre: Failing over lustre-MDT0000 [ 7760.621597] LustreError: 358005:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7760.626753] LustreError: 358005:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7760.739229] LustreError: 357500:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7760.783342] Lustre: server umount lustre-MDT0000 complete [ 7760.871987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7760.896673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7765.818310] LustreError: 355538:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7765.830725] LustreError: 355538:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 4 previous similar messages [ 7776.057634] LustreError: 355537:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7776.076647] LustreError: 355537:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 7 previous similar messages [ 7779.063698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7779.240526] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7779.503565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7779.510965] Lustre: Skipped 2 previous similar messages [ 7779.576604] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7782.203862] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7783.458524] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7784.945387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7784.969891] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 7784.998202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7790.501253] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7791.995339] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7795.310203] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7796.955132] Lustre: Failing over lustre-MDT0000 [ 7797.130622] LustreError: 359085:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7797.137543] LustreError: 359085:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7797.295188] Lustre: server umount lustre-MDT0000 complete [ 7797.592083] LustreError: 357500:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7797.602096] LustreError: 357500:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 4 previous similar messages [ 7800.289161] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.314534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7800.327551] Lustre: Skipped 3 previous similar messages [ 7815.892969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7816.002719] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7816.285708] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7818.042856] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7819.897464] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7821.299613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7821.301913] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7821.307701] Lustre: Skipped 2 previous similar messages [ 7821.350901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7826.244775] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7827.733792] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7836.641662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7836.645876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7836.654497] Lustre: Skipped 1 previous similar message [ 7836.661107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7836.667807] Lustre: Skipped 5 previous similar messages [ 7839.005969] LustreError: 360121:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7839.011712] LustreError: 360121:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7839.070984] Lustre: server umount lustre-MDT0000 complete [ 7841.763774] LustreError: 356576:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7841.777880] LustreError: 356576:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 16 previous similar messages [ 7842.591371] LustreError: 355523:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761292146 with bad export cookie 778552138755324857 [ 7842.593119] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7842.618596] LustreError: 355523:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7843.003516] Lustre: server umount lustre-MDT0001 complete [ 7857.547408] Lustre: server umount lustre-OST0000 complete [ 7874.972138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7875.415446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7875.418667] Lustre: Skipped 1 previous similar message [ 7879.120977] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7887.527756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7891.741439] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7897.824280] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7901.213320] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7907.509374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7913.680566] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7919.077167] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7920.120071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7930.341557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7930.348650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7930.359308] Lustre: Skipped 3 previous similar messages [ 7930.373121] Lustre: Skipped 1 previous similar message [ 7934.529229] LustreError: 363601:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7934.534512] LustreError: 363601:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7934.790950] Lustre: server umount lustre-OST0000 complete [ 7939.040184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7945.001949] Lustre: server umount lustre-MDT0000 complete [ 7945.698379] LustreError: 361352:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7945.716837] LustreError: 361352:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 7948.322539] LustreError: 363803:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761292252 with bad export cookie 778552138755325585 [ 7948.328706] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7948.332358] LustreError: 363803:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7948.833183] Lustre: server umount lustre-MDT0001 complete [ 7959.305447] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 7962.475829] Key type lgssc unregistered [ 7962.775878] LNet: 364538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7962.782402] LNetError: 364538:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7962.801477] LNet: Removed LNI 192.168.203.102@tcp [ 7963.651221] Key type .llcrypt unregistered [ 7963.652951] Key type ._llcrypt unregistered [ 7988.326710] Key type ._llcrypt registered [ 7988.338049] Key type .llcrypt registered [ 7988.436637] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 7989.461167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7989.476330] alg: No test for adler32 (adler32-zlib) [ 7990.510813] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 7990.692697] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 7992.335189] Key type lgssc registered [ 7993.228673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7999.783378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7999.813815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8001.347830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8005.030470] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8011.748816] LustreError: 366035:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8011.797400] LustreError: 366035:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 8014.506599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8015.020933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8018.715418] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8024.335406] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8028.141204] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8035.264454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8035.713579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8040.654316] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8045.340146] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8045.950921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8055.776950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8055.789521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8055.809457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8056.294754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8056.300082] Lustre: Skipped 1 previous similar message [ 8056.993402] LustreError: 368281:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8057.051848] Lustre: server umount lustre-MDT0000 complete [ 8060.397482] LustreError: 366016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761292364 with bad export cookie 7845702140423003685 [ 8060.404666] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8060.417337] LustreError: 366016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8060.546461] LustreError: 368483:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8060.551265] LustreError: 368483:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8060.761983] Lustre: server umount lustre-MDT0001 complete [ 8074.340405] LustreError: 368686:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8074.348969] LustreError: 368686:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8074.623504] Lustre: server umount lustre-OST0000 complete [ 8088.377176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8089.156865] LustreError: 369415:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8089.254510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8092.992827] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8094.688453] LustreError: 369415:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8099.811044] LustreError: 369415:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8102.318411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8102.766756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8106.861280] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8112.594522] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8116.054487] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8123.868132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8124.463969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8130.714693] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8134.725340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 8137.322522] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8154.082557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8154.101896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8154.130747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8154.133228] Lustre: Skipped 3 previous similar messages [ 8156.282184] LustreError: 371766:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8156.286304] LustreError: 371766:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8156.454494] Lustre: server umount lustre-MDT0000 complete [ 8160.238915] LustreError: 369411:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8160.263086] LustreError: 369411:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 8160.404821] LustreError: 370135:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761292464 with bad export cookie 7845702140423004798 [ 8160.414386] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8160.416096] LustreError: 370135:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8160.626029] LustreError: 371967:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8160.649747] LustreError: 371967:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8161.052920] Lustre: server umount lustre-MDT0001 complete [ 8175.006660] LustreError: 372170:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8175.014660] LustreError: 372170:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8175.312337] Lustre: server umount lustre-OST0000 complete [ 8191.735747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8192.213300] LustreError: 372999:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8192.291906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8196.830335] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8206.281587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8210.654290] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8216.313348] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8220.043253] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8227.604626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8228.030225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8228.033434] Lustre: Skipped 1 previous similar message [ 8233.282758] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8237.241281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 8238.954474] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8242.663497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8242.680522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8242.680531] Lustre: Skipped 3 previous similar messages [ 8242.686886] Lustre: Skipped 3 previous similar messages [ 8244.737190] LustreError: 375247:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8244.747893] LustreError: 375247:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8244.954624] Lustre: server umount lustre-OST0000 complete [ 8252.900893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8252.901690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8252.906300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8252.906649] Lustre: Skipped 1 previous similar message [ 8252.908610] Lustre: Skipped 1 previous similar message [ 8255.369316] Lustre: server umount lustre-MDT0000 complete [ 8258.018901] LustreError: 374132:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8258.027419] LustreError: 374132:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 8259.560194] LustreError: 372978:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761292563 with bad export cookie 7845702140423005813 [ 8259.571518] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8259.574263] LustreError: 372978:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8259.966297] Lustre: server umount lustre-MDT0001 complete [ 8270.631518] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 8273.234673] Key type lgssc unregistered [ 8273.499415] LNet: 376183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8273.509262] LNetError: 376183:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8274.542609] LNet: Removed LNI 192.168.203.102@tcp [ 8275.477649] Key type .llcrypt unregistered [ 8275.480750] Key type ._llcrypt unregistered [ 8303.569283] Key type ._llcrypt registered [ 8303.572289] Key type .llcrypt registered [ 8303.670085] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 8304.782568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8304.971720] alg: No test for adler32 (adler32-zlib) [ 8306.016667] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 8306.201332] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 8307.887197] Key type lgssc registered [ 8309.152724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8317.897136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8317.927086] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8317.934470] LustreError: 377604:0:(osd_handler.c:8747:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8317.964484] LustreError: 377604:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8317.978844] LustreError: 377604:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8317.989226] LustreError: 377604:0:(tgt_mount.c:2450:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8317.994799] LustreError: 377604:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8325.010513] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8325.015673] LustreError: 378003:0:(osd_handler.c:8747:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8325.031791] LustreError: 378003:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8325.036275] LustreError: 378003:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8325.040075] LustreError: 378003:0:(tgt_mount.c:2450:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8325.061856] LustreError: 378003:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8344.600122] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 8347.792501] Key type lgssc unregistered [ 8348.075652] LNet: 378988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8348.086616] LNetError: 378988:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8348.101711] LNet: Removed LNI 192.168.203.102@tcp [ 8349.018488] Key type .llcrypt unregistered [ 8349.020174] Key type ._llcrypt unregistered [ 8367.221489] Key type ._llcrypt registered [ 8367.229952] Key type .llcrypt registered [ 8367.377618] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 8380.849565] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 8381.730760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8381.750554] alg: No test for adler32 (adler32-zlib) [ 8382.992475] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 8383.222395] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 8384.898747] Key type lgssc registered [ 8386.416875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8395.901289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8402.479125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8408.409619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8414.821850] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8422.060903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8422.109039] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8422.127983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8423.386951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8423.415284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8423.502917] Lustre: lustre-MDT0000: new disk, initializing [ 8423.589206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8423.610990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8427.483784] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8438.170716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8438.257955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8438.382688] Lustre: 382125:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8438.454720] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8438.457429] Lustre: Skipped 1 previous similar message [ 8438.576802] Lustre: lustre-MDT0001: new disk, initializing [ 8438.693748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8438.726226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8438.739062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8438.838829] Lustre: 382921:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 8438.853896] Lustre: 382921:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 8438.869677] Lustre: 382921:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 8438.878545] Lustre: 382921:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 8438.904548] Lustre: 382921:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 8438.918712] Lustre: 382921:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 8442.854493] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8450.238081] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8453.995244] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8460.082559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8460.145954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8460.380988] Lustre: lustre-OST0000: new disk, initializing [ 8460.388499] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8460.464278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8465.944589] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8469.050348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8469.061760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8469.173167] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8472.941757] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8477.282299] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8477.498517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8481.405668] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8481.648486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8484.833382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8484.840111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8484.854814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8484.858084] Lustre: Skipped 1 previous similar message [ 8484.866231] Lustre: Skipped 1 previous similar message [ 8489.494292] LustreError: 385066:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8489.745380] Lustre: server umount lustre-OST0000 complete [ 8495.074368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8495.079066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8495.097920] LustreError: Skipped 1 previous similar message [ 8495.099232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8495.158550] Lustre: Skipped 1 previous similar message [ 8499.731134] LustreError: 385268:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8499.734379] LustreError: 385268:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8499.826406] Lustre: server umount lustre-MDT0000 complete [ 8503.619550] LustreError: 383882:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761292807 with bad export cookie 14125857127024962112 [ 8503.622313] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8503.655587] LustreError: 383882:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8503.871747] LustreError: 385469:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8503.883914] LustreError: 385469:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8504.167803] Lustre: server umount lustre-MDT0001 complete [ 8513.467372] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8515.055909] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:00:17 (1761292817) [ 8521.247782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8521.748889] LustreError: 386186:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8521.766953] LustreError: 386186:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 8521.811295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8525.489662] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8526.818242] LustreError: 386186:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8530.914452] LustreError: 386186:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8534.579964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8534.929415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8539.272733] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8544.939577] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8548.169985] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8555.348212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8555.834399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8561.882639] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8567.194549] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8574.254931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8574.359969] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8574.548672] Lustre: lustre-OST0001: new disk, initializing [ 8574.553802] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8574.703962] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8580.350253] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8583.316198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8583.328571] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8583.425749] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8587.904604] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8593.888915] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8593.892701] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8593.895958] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8593.915483] Lustre: Skipped 1 previous similar message [ 8593.918260] Lustre: Skipped 1 previous similar message [ 8597.483533] LustreError: 389400:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 8597.485971] LustreError: 389400:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8597.571941] Lustre: server umount lustre-OST0001 complete [ 8599.022317] LustreError: 387788:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8599.037492] LustreError: 387788:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 8604.131926] LustreError: 387788:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8604.153726] LustreError: 387788:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 8614.382216] LustreError: 387786:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8614.385374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8614.413662] LustreError: 387786:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 8614.425779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8614.429337] Lustre: Skipped 3 previous similar messages [ 8619.492298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8619.504394] Lustre: Skipped 1 previous similar message [ 8627.167427] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8627.223409] LustreError: 389625:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8627.238062] LustreError: 389625:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8627.481110] Lustre: server umount lustre-OST0000 complete [ 8632.288289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8632.298353] LustreError: Skipped 1 previous similar message [ 8632.307385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8632.315066] Lustre: Skipped 1 previous similar message [ 8632.333521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8632.336261] Lustre: Skipped 2 previous similar messages [ 8645.087164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8645.186262] LustreError: 389828:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8645.199479] LustreError: 389828:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8645.297969] Lustre: server umount lustre-MDT0000 complete [ 8648.548819] LustreError: 386168:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761292952 with bad export cookie 14125857127024963036 [ 8648.568520] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8655.020556] Lustre: server umount lustre-MDT0001 complete [ 8663.899928] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 8666.143217] Key type lgssc unregistered [ 8666.365111] LNet: 390565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8666.376113] LNetError: 390565:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8667.434822] LNet: Removed LNI 192.168.203.102@tcp [ 8668.363107] Key type .llcrypt unregistered [ 8668.365159] Key type ._llcrypt unregistered [ 8698.405408] Key type ._llcrypt registered [ 8698.407199] Key type .llcrypt registered [ 8698.523412] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 8712.028254] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 8713.027626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8713.107444] alg: No test for adler32 (adler32-zlib) [ 8714.125195] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 8714.329868] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 8715.983193] Key type lgssc registered [ 8716.966585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8724.426652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8729.418784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8734.584566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8740.011391] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8746.496849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8746.560189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8746.580713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8747.923727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8747.992267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8748.076546] Lustre: lustre-MDT0000: new disk, initializing [ 8748.196568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8748.214296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8752.141877] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8762.347460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8762.419197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8762.518957] Lustre: 394534:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8762.600422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8762.608249] Lustre: Skipped 1 previous similar message [ 8762.613171] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8762.698450] Lustre: lustre-MDT0001: new disk, initializing [ 8762.809530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8762.883153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8762.903867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8766.878806] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8774.380544] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8777.949306] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8783.498506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8783.602938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8783.854551] Lustre: lustre-OST0000: new disk, initializing [ 8783.862348] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8783.952547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8788.769673] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8790.574437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8790.581093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8790.719206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8795.817879] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8799.814472] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8799.991231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8804.263630] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8804.506501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8806.384555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8806.401723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8811.491515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8811.496512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8811.506813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8812.473029] LustreError: 397473:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8812.654185] Lustre: server umount lustre-OST0000 complete [ 8816.609458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8816.636485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8816.645628] Lustre: Skipped 2 previous similar messages [ 8821.732784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8821.740283] Lustre: Skipped 1 previous similar message [ 8822.238072] LustreError: 397675:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8822.245576] LustreError: 397675:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8822.328713] Lustre: server umount lustre-MDT0000 complete [ 8826.072225] LustreError: 394526:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761293130 with bad export cookie 5485646623064507942 [ 8826.075555] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8826.079925] LustreError: 394526:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8826.225912] LustreError: 397876:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8826.233880] LustreError: 397876:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8826.562989] Lustre: server umount lustre-MDT0001 complete [ 8835.093718] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 04:05:37 (1761293137) [ 8843.311238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8850.899315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8851.262535] Lustre: lustre-MDT0000: new disk, initializing [ 8851.398292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8854.895823] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8856.549494] LustreError: 399143:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8856.573296] LustreError: 399143:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 8861.665026] LustreError: 399143:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8862.736610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8863.137404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8866.447222] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8871.014070] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8873.923986] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8878.560791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8878.564296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8878.564950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8878.579369] Lustre: Skipped 2 previous similar messages [ 8881.831768] LustreError: 400491:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8881.835256] LustreError: 400491:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8881.911315] Lustre: server umount lustre-MDT0000 complete [ 8883.680584] LustreError: 399143:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8883.703284] LustreError: 399143:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 8885.058849] LustreError: 399124:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761293189 with bad export cookie 5485646623064508866 [ 8885.059700] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8885.068809] LustreError: 399124:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8885.467802] Lustre: server umount lustre-MDT0001 complete [ 8894.304788] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:06:36 (1761293196) [ 8901.096156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8901.816585] LustreError: 401411:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8901.923177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8905.549791] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8907.237674] LustreError: 401411:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8912.819230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8917.499919] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8922.300643] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8926.041451] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8932.720883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8933.000501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8933.003413] Lustre: Skipped 1 previous similar message [ 8938.168585] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8942.832777] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8953.979312] Lustre: Permanently deactivating lustre-OST0000 [ 8953.995466] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8962.617450] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8963.549506] LustreError: 404059:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8963.558811] LustreError: 404059:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 8963.562867] LustreError: 404059:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8966.278430] LustreError: 404108:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 8966.282803] LustreError: 404108:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8974.469240] LustreError: 404205:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8974.474763] LustreError: 404205:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8974.674849] Lustre: server umount lustre-OST0000 complete [ 8976.880927] LustreError: 404357:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 8976.892909] LustreError: 404357:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8979.938706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8979.943822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8979.969197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8979.973722] Lustre: Skipped 1 previous similar message [ 8988.665859] LustreError: 404454:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8988.668496] LustreError: 404454:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8988.748521] Lustre: server umount lustre-MDT0000 complete [ 8992.005947] LustreError: 401392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761293296 with bad export cookie 5485646623064509419 [ 8992.017286] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8992.023136] LustreError: 401392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9002.689123] Lustre: server umount lustre-MDT0001 complete [ 9011.795308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9015.381510] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9018.095137] LustreError: 405501:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 9023.945509] Lustre: 405842:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.102@tcp [ 9026.073534] LustreError: 405939:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9026.077201] LustreError: 405939:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9026.173391] Lustre: server umount MGS complete [ 9036.728695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9037.012758] LustreError: 406567:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9037.028955] LustreError: 406567:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 9037.033770] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9037.040100] Lustre: Skipped 1 previous similar message [ 9037.094484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9040.326266] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9048.022779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9048.286262] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9051.685516] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9057.086624] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9060.640501] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9062.356352] Lustre: Permanently reactivating lustre-OST0000 [ 9062.375141] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9062.378914] Lustre: Skipped 2 previous similar messages [ 9085.920276] LustreError: 408025:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 9092.345847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9097.332710] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9101.714318] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9105.894785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9105.910037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9105.934384] Lustre: Skipped 1 previous similar message [ 9105.944822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9105.947877] Lustre: Skipped 4 previous similar messages [ 9111.293186] LustreError: 409593:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9111.295869] LustreError: 409593:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 9111.504241] Lustre: server umount lustre-OST0000 complete [ 9115.108162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9115.114121] LustreError: Skipped 1 previous similar message [ 9120.875860] Lustre: server umount lustre-MDT0000 complete [ 9122.275636] LustreError: 408124:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9122.287729] LustreError: 408124:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 9123.689293] LustreError: 406548:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761293427 with bad export cookie 5485646623064510378 [ 9123.692980] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9123.695668] LustreError: 406548:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9124.068227] Lustre: server umount lustre-MDT0001 complete [ 9131.920159] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 9134.221295] Key type lgssc unregistered [ 9134.481487] LNet: 410531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9134.488584] LNetError: 410531:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9134.498169] LNet: Removed LNI 192.168.203.102@tcp [ 9135.203723] Key type .llcrypt unregistered [ 9135.205743] Key type ._llcrypt unregistered [ 9150.102904] Key type ._llcrypt registered [ 9150.104958] Key type .llcrypt registered [ 9150.190091] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 9161.041191] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 9161.659818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9161.686136] alg: No test for adler32 (adler32-zlib) [ 9162.697345] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 9162.965304] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 9164.663193] Key type lgssc registered [ 9165.684850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9173.321193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9178.995085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9183.969308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9188.863156] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9193.389169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9193.436879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9193.446773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9194.590496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9194.611298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9194.666493] Lustre: lustre-MDT0000: new disk, initializing [ 9194.731189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9194.745962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9197.302566] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9204.781171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9204.840899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9204.907121] Lustre: 413640:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9204.964076] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9204.967481] Lustre: Skipped 1 previous similar message [ 9205.024385] Lustre: lustre-MDT0001: new disk, initializing [ 9205.125016] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9205.152253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9205.157872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9205.240800] Lustre: 414438:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 9205.247550] Lustre: 414438:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 9205.256430] Lustre: 414438:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 9205.263614] Lustre: 414438:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 9205.269675] Lustre: 414438:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 9205.282365] Lustre: 414438:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 9207.666841] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9212.805143] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9215.195476] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9220.176604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9220.244662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9220.418916] Lustre: lustre-OST0000: new disk, initializing [ 9220.424548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9220.507923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9222.452796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9222.470579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9222.593791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9224.349967] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9229.571743] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9232.600592] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9232.767472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9236.131717] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9236.313706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9237.985250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9237.990153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9238.000807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9238.004155] Lustre: Skipped 1 previous similar message [ 9243.108111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9243.114726] Lustre: Skipped 1 previous similar message [ 9248.227687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9248.230877] Lustre: Skipped 1 previous similar message [ 9252.319179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9252.350083] LustreError: 416582:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9252.462024] Lustre: server umount lustre-OST0000 complete [ 9256.416343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9256.423679] LustreError: Skipped 1 previous similar message [ 9256.426213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9256.436710] Lustre: Skipped 1 previous similar message [ 9256.440848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9258.465414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9263.585110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9263.587624] Lustre: Skipped 2 previous similar messages [ 9269.216846] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9269.312576] LustreError: 416787:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9269.316641] LustreError: 416787:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9269.356710] Lustre: server umount lustre-MDT0000 complete [ 9271.815497] LustreError: 413632:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761293575 with bad export cookie 8773879860346449169 [ 9271.823423] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9271.829589] LustreError: 413632:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9271.931756] LustreError: 416989:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9271.934432] LustreError: 416989:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9272.122348] Lustre: server umount lustre-MDT0001 complete [ 9279.244160] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 9280.713144] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:13:02 (1761293582) [ 9285.886922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9286.350563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9289.848516] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9291.746867] LustreError: 417708:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9291.754452] LustreError: 417708:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 9296.574911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9296.836929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9300.108474] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9304.307690] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9307.231470] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9312.668817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9312.954397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9317.068292] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9320.888502] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9322.326183] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 9333.758638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 9337.831617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9337.834965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9337.837630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9337.847818] Lustre: Skipped 1 previous similar message [ 9337.854353] Lustre: Skipped 2 previous similar messages [ 9343.415195] LustreError: 420172:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9343.423356] LustreError: 420172:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9343.551457] Lustre: server umount lustre-OST0000 complete [ 9348.064103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9348.079833] Lustre: Skipped 1 previous similar message [ 9358.307497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9358.311627] Lustre: Skipped 8 previous similar messages [ 9361.375185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9361.465195] LustreError: 420374:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9361.473787] LustreError: 420374:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9361.539317] Lustre: server umount lustre-MDT0000 complete [ 9363.424882] LustreError: 417708:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9363.438171] LustreError: 417708:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 9363.956763] LustreError: 417688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761293668 with bad export cookie 8773879860346450093 [ 9363.962026] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9363.962940] LustreError: 417688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9364.180772] Lustre: server umount lustre-MDT0001 complete [ 9371.111304] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 9372.988427] Key type lgssc unregistered [ 9373.170310] LNet: 421110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9373.176368] LNetError: 421110:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9374.246284] LNet: Removed LNI 192.168.203.102@tcp [ 9374.745615] Key type .llcrypt unregistered [ 9374.747409] Key type ._llcrypt unregistered [ 9391.541263] Key type ._llcrypt registered [ 9391.542669] Key type .llcrypt registered [ 9391.629458] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 9391.836511] Key type .llcrypt unregistered [ 9391.839828] Key type ._llcrypt unregistered [ 9402.457442] Key type ._llcrypt registered [ 9402.459896] Key type .llcrypt registered [ 9402.540935] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 9403.038516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9403.083212] alg: No test for adler32 (adler32-zlib) [ 9404.013530] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 9404.136996] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 9405.752734] Key type lgssc registered [ 9406.432231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9411.081899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9411.098345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9412.380480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9414.600538] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9420.416143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9420.626957] LustreError: 423260:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9420.634190] LustreError: 423260:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 9420.650675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9422.689703] LustreError: 423260:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9422.700888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9422.702582] LustreError: 423260:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 9424.490866] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9427.941326] LustreError: 423260:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9427.958415] LustreError: 423260:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 9428.306175] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9432.872621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9433.093686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9435.335149] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9442.596752] Lustre: 423257:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 9442.605309] Lustre: 423257:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 9442.613379] Lustre: 423257:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 9442.617382] Lustre: 423257:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 9442.621545] Lustre: 423257:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 9442.625846] Lustre: 423257:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 9448.419139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9448.420220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9448.420764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9448.431101] Lustre: Skipped 1 previous similar message [ 9450.990080] LustreError: 425317:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 9451.095785] Lustre: server umount lustre-OST0000 complete [ 9455.076599] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 9455.080161] LustreError: Skipped 1 previous similar message [ 9455.081944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9455.090738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9455.095368] Lustre: Skipped 1 previous similar message [ 9455.765540] LustreError: 425519:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9455.769399] LustreError: 425519:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9455.827018] Lustre: server umount lustre-MDT0000 complete [ 9458.356560] LustreError: 423241:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761293762 with bad export cookie 2920078683880556377 [ 9458.364429] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9458.366565] LustreError: 423241:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9458.469735] LustreError: 425719:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9458.472448] LustreError: 425719:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9458.608996] Lustre: server umount lustre-MDT0001 complete [ 9465.254910] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 9467.352331] Key type lgssc unregistered [ 9467.579197] LNet: 426250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9467.586488] LNetError: 426250:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9467.603319] LNet: Removed LNI 192.168.203.102@tcp [ 9468.197728] Key type .llcrypt unregistered [ 9468.199258] Key type ._llcrypt unregistered [ 9482.837134] Key type ._llcrypt registered [ 9482.838788] Key type .llcrypt registered [ 9482.898388] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 9483.461108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9483.492467] alg: No test for adler32 (adler32-zlib) [ 9484.429528] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 9484.558874] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 9486.167212] Key type lgssc registered [ 9486.850731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9492.102047] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9492.116138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9507.615305] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9512.735271] LustreError: 427598:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.102@tcp: failed processing log, type 4: rc = -110 [ 9539.359233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9543.364200] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9546.748837] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9550.791754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9551.138398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9551.198803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9553.589392] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9559.108767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9559.316867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9561.565939] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9565.082469] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9567.409065] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9570.107141] Lustre: 429242:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 9570.110353] Lustre: 429242:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 9570.112750] Lustre: 429242:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 9570.115798] Lustre: 429242:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 9570.118363] Lustre: 429242:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 9570.121760] Lustre: 429242:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 9574.885911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9574.886500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9574.892093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9574.898403] Lustre: Skipped 1 previous similar message [ 9574.904028] Lustre: Skipped 1 previous similar message [ 9578.616534] LustreError: 429879:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 9578.770343] Lustre: server umount lustre-OST0000 complete [ 9581.489960] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 9581.493113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9581.501844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9585.120996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9585.121415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9587.677398] LustreError: 430080:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 9587.680750] LustreError: 430080:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9587.773384] Lustre: server umount lustre-MDT0000 complete [ 9590.240478] LustreError: 428522:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9590.253071] LustreError: 428522:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 9590.582775] LustreError: 427607:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761293894 with bad export cookie 10888013475504309876 [ 9590.584796] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9590.587528] LustreError: 427607:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9590.673559] LustreError: 430282:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9590.676829] LustreError: 430282:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9590.848586] Lustre: server umount lustre-MDT0001 complete [ 9597.900291] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 9600.027325] Key type lgssc unregistered [ 9600.235433] LNet: 430814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9600.246708] LNetError: 430814:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9600.267411] LNet: Removed LNI 192.168.203.102@tcp [ 9600.867886] Key type .llcrypt unregistered [ 9600.874863] Key type ._llcrypt unregistered [ 9618.110750] Key type ._llcrypt registered [ 9618.113637] Key type .llcrypt registered [ 9618.177704] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 9618.739562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9618.830480] alg: No test for adler32 (adler32-zlib) [ 9619.784714] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 9619.947389] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 9621.575257] Key type lgssc registered [ 9622.190801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9626.983228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9627.001129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9628.340773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9630.773746] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9636.460345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9636.700807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9639.208970] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9643.078861] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9645.421939] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9650.022347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9650.268995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9653.286128] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9653.360232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9656.684977] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9662.432628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9662.438553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9662.446675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9663.320639] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.2@tcp (stopping) [ 9663.465694] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9665.743405] LustreError: 434447:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9665.816120] Lustre: server umount lustre-MDT0000 complete [ 9666.997214] LustreError: 433823:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.2@tcp arrived at 1761293971 with bad export cookie 14776054883851191241 [ 9667.004916] LustreError: 433823:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9667.970856] LustreError: 433823:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761293972 with bad export cookie 14776054883851190345 [ 9667.976937] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9668.576979] LustreError: 432200:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9668.577180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9668.591206] LustreError: 432200:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 9668.608429] Lustre: Skipped 1 previous similar message [ 9668.613551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9668.617466] Lustre: Skipped 3 previous similar messages [ 9670.009296] LustreError: 434650:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9670.018753] LustreError: 434650:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9670.127804] Lustre: server umount lustre-OST0000 complete [ 9672.610735] LustreError: 432200:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9672.617907] LustreError: 432200:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 9675.743491] LustreError: 432917:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d14725a9 x1846850537678720/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9675.764900] LustreError: 435003:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9675.774455] LustreError: 435003:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9675.911894] Lustre: server umount lustre-MDT0001 complete [ 9682.113934] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 9683.727576] Key type lgssc unregistered [ 9683.900913] LNet: 435535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9683.909062] LNetError: 435535:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9683.922506] LNet: Removed LNI 192.168.203.102@tcp [ 9684.368207] Key type .llcrypt unregistered [ 9684.370176] Key type ._llcrypt unregistered [ 9698.721533] Key type ._llcrypt registered [ 9698.723648] Key type .llcrypt registered [ 9698.797613] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 9699.199656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9699.244121] alg: No test for adler32 (adler32-zlib) [ 9700.190456] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 9700.356679] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 9701.992421] Key type lgssc registered [ 9702.746884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9707.704880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9707.736233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9709.077116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9711.316694] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9715.703331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9715.919366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9717.840387] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9720.660886] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9722.597571] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9726.415615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9726.619774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9730.111754] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9733.036276] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9734.826346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9744.700076] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.2@tcp (stopping) [ 9744.866507] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9744.869745] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9745.892202] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9745.892326] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9745.901909] Lustre: Skipped 1 previous similar message [ 9746.834192] LustreError: 439162:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9746.886884] Lustre: server umount lustre-MDT0001 complete [ 9750.496710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9750.505415] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9750.515429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9751.009984] LustreError: 436916:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9751.017308] LustreError: 436916:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 9755.694300] LustreError: 439367:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9755.698176] LustreError: 439367:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9755.791546] Lustre: server umount lustre-OST0000 complete [ 9756.129161] LustreError: 436916:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9757.947868] LustreError: 439568:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9757.954794] LustreError: 439568:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9758.089095] Lustre: server umount lustre-MDT0000 complete [ 9764.943572] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 9766.301544] Key type lgssc unregistered [ 9766.463969] LNet: 440251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9766.467976] LNetError: 440251:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9766.481395] LNet: Removed LNI 192.168.203.102@tcp [ 9766.866342] Key type .llcrypt unregistered [ 9766.868157] Key type ._llcrypt unregistered [ 9780.347326] Key type ._llcrypt registered [ 9780.348942] Key type .llcrypt registered [ 9780.398826] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 9788.392335] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 9788.807202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9788.814938] alg: No test for adler32 (adler32-zlib) [ 9789.698409] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 9789.812737] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 9791.423263] Key type lgssc registered [ 9792.036482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9797.251800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9800.527166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9803.403459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9807.046569] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9812.207582] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 9816.852121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9820.745241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9820.778440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9820.793377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9821.910366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9821.932776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9821.976291] Lustre: lustre-MDT0000: new disk, initializing [ 9822.026875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9822.046679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9824.155711] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9829.459032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9833.034825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9833.081414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9833.232377] Lustre: lustre-OST0000: new disk, initializing [ 9833.238444] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9833.247082] Lustre: Skipped 1 previous similar message [ 9833.289749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9836.522313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9836.535656] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9836.556047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9836.807437] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9841.329463] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9847.775036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9851.811649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9851.861840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9851.927140] Lustre: 444216:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9851.972981] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9852.017843] Lustre: lustre-MDT0001: new disk, initializing [ 9852.106581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9852.137262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 9852.145332] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 9852.186862] Lustre: 446459:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 9852.191347] Lustre: 446459:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 9852.195058] Lustre: 446459:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 9852.199357] Lustre: 446459:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 9852.209854] Lustre: 446459:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 9852.213850] Lustre: 446459:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 9854.285984] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9859.788288] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9859.903441] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9862.313990] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9862.451080] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9866.213241] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9866.215247] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9866.224056] Lustre: Skipped 1 previous similar message [ 9869.129034] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.2@tcp (stopping) [ 9869.133346] Lustre: Skipped 1 previous similar message [ 9871.329953] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9871.333088] Lustre: Skipped 1 previous similar message [ 9871.764497] LustreError: 447638:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9871.817360] Lustre: server umount lustre-MDT0001 complete [ 9881.574497] LustreError: 444227:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9881.585771] LustreError: 444227:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 4 previous similar messages [ 9886.688190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9886.691796] LustreError: 444227:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9886.697899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9891.811477] LustreError: 444227:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9891.812885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9892.220115] LustreError: 447909:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 9892.230739] LustreError: 447909:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9892.338818] Lustre: server umount lustre-OST0000 complete [ 9909.215135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9909.286622] LustreError: 448110:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 9909.295760] LustreError: 448110:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9909.445310] Lustre: server umount lustre-MDT0000 complete [ 9916.861654] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 9918.601489] Key type lgssc unregistered [ 9918.816829] LNet: 448795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9918.820721] LNetError: 448795:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9918.831569] LNet: Removed LNI 192.168.203.102@tcp [ 9919.330376] Key type .llcrypt unregistered [ 9919.332241] Key type ._llcrypt unregistered [ 9939.686961] Key type ._llcrypt registered [ 9939.688788] Key type .llcrypt registered [ 9939.747424] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 9939.949671] Key type .llcrypt unregistered [ 9939.952092] Key type ._llcrypt unregistered [ 9949.697140] Key type ._llcrypt registered [ 9949.699820] Key type .llcrypt registered [ 9949.773805] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 9950.440569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9950.469654] alg: No test for adler32 (adler32-zlib) [ 9951.407852] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 9951.574846] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 9953.215630] Key type lgssc registered [ 9953.882965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9958.863557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9962.711174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9966.650669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9970.691831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9970.739258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9970.765751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9971.895224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9971.912654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9971.957458] Lustre: lustre-MDT0000: new disk, initializing [ 9972.013092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9972.029040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9974.285792] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9980.637669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9980.676642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9980.730602] Lustre: 452119:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9980.772624] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9980.775824] Lustre: Skipped 1 previous similar message [ 9980.816497] Lustre: lustre-MDT0001: new disk, initializing [ 9980.862899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9980.885240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9980.889341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9980.930691] Lustre: 452916:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 9980.935621] Lustre: 452916:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 9980.939582] Lustre: 452916:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 9980.944088] Lustre: 452916:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 9980.952122] Lustre: 452916:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 9980.959839] Lustre: 452916:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 9982.850979] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9987.615886] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9989.558989] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9993.310022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9993.344827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9993.494478] Lustre: lustre-OST0000: new disk, initializing [ 9993.497557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9993.550344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9994.022205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9994.027996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9994.097855] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9996.195330] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10000.197948] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10004.448957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10004.451240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10004.454693] Lustre: Skipped 2 previous similar messages [10004.457121] Lustre: Skipped 1 previous similar message [10009.428282] LustreError: 454564:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10009.498023] Lustre: server umount lustre-MDT0000 complete [10011.677489] LustreError: 452110:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761294315 with bad export cookie 16053206648069009509 [10011.678659] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10011.686271] LustreError: 452110:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10011.780024] LustreError: 454766:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10011.782839] LustreError: 454766:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10011.906549] Lustre: server umount lustre-MDT0001 complete [10024.193288] LustreError: 454967:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10024.198129] LustreError: 454967:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10024.322266] Lustre: server umount lustre-OST0000 complete [10030.640203] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:25:33 (1761294333) [10031.189178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10035.167752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10035.437430] LustreError: 455794:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10035.452152] LustreError: 455794:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 4 previous similar messages [10035.514312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10037.813556] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10040.801931] LustreError: 455794:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10042.375508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10042.570057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10044.658073] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10047.819886] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10049.851923] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10053.850340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10053.899234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10054.012725] Lustre: Found index 0 for lustre-OST0000, updating log [10054.151664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10057.449524] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10060.352581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10060.684714] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10065.377096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10065.382133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10065.397563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10066.467910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10068.515806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10068.526390] Lustre: Skipped 1 previous similar message [10069.174146] LustreError: 458052:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10069.177401] LustreError: 458052:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10069.253456] Lustre: server umount lustre-OST0000 complete [10070.737082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10073.569196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10073.569666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10073.570435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10073.570442] Lustre: Skipped 1 previous similar message [10073.587022] Lustre: Skipped 1 previous similar message [10084.833105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10084.836234] Lustre: Skipped 5 previous similar messages [10085.855398] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10085.930193] LustreError: 458306:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10085.936943] LustreError: 458306:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10085.982158] Lustre: server umount lustre-MDT0000 complete [10088.162916] LustreError: 458307:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761294392 with bad export cookie 16053206648069010937 [10088.164723] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10088.170662] LustreError: 458307:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10088.369815] Lustre: server umount lustre-MDT0001 complete [10092.606691] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:26:35 (1761294395) [10093.141112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10093.700546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10094.221586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10094.817947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10095.353540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10095.904110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10096.513398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10097.033807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10097.570655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10098.209264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10098.742860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10099.312149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10099.801595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10100.269565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10100.716942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10101.198965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10101.638437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10102.112576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10102.600077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10103.042209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10103.513063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10104.038515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10104.511707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10105.035948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10105.498769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10105.935982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10106.384258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10106.814129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10107.262496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10107.716870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10108.171529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10108.624175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10109.165784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10109.649761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10110.147944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10110.633103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10111.112834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10111.626212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10112.208970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10112.732510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10113.529220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10114.048986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10114.573953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10115.077579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10115.627843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10116.120261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10116.664433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10117.211261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10117.751687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10118.332694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10118.927673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10122.729596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10122.941192] LustreError: 461894:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10122.977313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10125.014691] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10128.356492] LustreError: 461894:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10129.302464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10131.642188] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10134.981370] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10137.035358] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10141.204952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10141.275381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10141.419229] Lustre: Found index 0 for lustre-OST0000, updating log [10145.110672] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10148.028387] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10149.153107] LustreError: 464165:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10149.156279] LustreError: 464165:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10149.252419] Lustre: server umount lustre-OST0000 complete [10154.976546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10154.980133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10154.988910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10157.531801] Lustre: server umount lustre-MDT0000 complete [10159.503566] LustreError: 461874:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761294463 with bad export cookie 16053206648069012015 [10159.504072] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10159.510982] LustreError: 461874:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10159.732858] Lustre: server umount lustre-MDT0001 complete [10161.245414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10169.120977] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [10172.991964] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [10177.093320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10180.014424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10182.784880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10185.586899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10188.475262] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 04:28:11 (1761294491) [10195.427773] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [10196.759507] Key type lgssc unregistered [10204.066967] LNet: 468128:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10204.071200] LNetError: 468128:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10319.783442] LNet: Removed LNI 192.168.203.102@tcp [10320.154857] Key type .llcrypt unregistered [10320.156585] Key type ._llcrypt unregistered [10328.120098] Key type ._llcrypt registered [10328.121469] Key type .llcrypt registered [10328.181106] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [10328.641884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10328.695951] alg: No test for adler32 (adler32-zlib) [10329.590792] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [10329.699897] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [10331.319745] Key type lgssc registered [10331.789931] Lustre: Echo OBD driver; http://www.lustre.org/ [10333.705084] LNet: Added LNI 192.168.203.102@tcp100 [8/256/0/180] [10334.380963] Lustre: DEBUG MARKER: SKIP: conf-sanity test_73c unable to configure net [10342.402707] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [10343.757474] Key type lgssc unregistered [10343.953709] LNet: 469996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10343.958615] LNetError: 469996:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10343.962048] LNet: Removed LNI 192.168.203.102@tcp [10343.969526] LNet: Removed LNI 192.168.203.102@tcp100 [10344.327561] Key type .llcrypt unregistered [10344.330102] Key type ._llcrypt unregistered [10355.470349] Key type ._llcrypt registered [10355.471841] Key type .llcrypt registered [10355.529977] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [10363.127544] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [10363.593542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10363.621495] alg: No test for adler32 (adler32-zlib) [10364.532829] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [10364.669770] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [10366.287163] Key type lgssc registered [10366.847985] Lustre: Echo OBD driver; http://www.lustre.org/ [10371.736741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10375.217814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10378.477213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10381.525345] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10385.242339] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 04:31:27 (1761294687) [10395.694917] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [10397.388115] Key type lgssc unregistered [10397.604672] LNet: 474009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10397.608891] LNetError: 474009:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10397.625484] LNet: Removed LNI 192.168.203.102@tcp [10398.084188] Key type .llcrypt unregistered [10398.085690] Key type ._llcrypt unregistered [10407.688283] Key type ._llcrypt registered [10407.689681] Key type .llcrypt registered [10407.750402] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [10408.155803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10408.209082] alg: No test for adler32 (adler32-zlib) [10409.119661] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [10409.241950] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [10410.863200] Key type lgssc registered [10411.390640] Lustre: Echo OBD driver; http://www.lustre.org/ [10416.524963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10416.552971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10416.567394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10417.693365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10417.718726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10417.764234] Lustre: lustre-MDT0000: new disk, initializing [10417.852121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10417.865444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10419.964352] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10426.023566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10426.063620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10426.128794] Lustre: 475182:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10426.178791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10426.181286] Lustre: Skipped 1 previous similar message [10426.223113] Lustre: lustre-MDT0001: new disk, initializing [10426.279392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10426.315043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10426.326318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10426.371095] Lustre: 475978:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [10426.380136] Lustre: 475978:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [10426.383185] Lustre: 475978:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [10426.386991] Lustre: 475978:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [10426.394026] Lustre: 475978:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [10426.398968] Lustre: 475978:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [10428.490566] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10432.662411] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10434.363950] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10437.955217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10437.993986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10438.160881] Lustre: lustre-OST0000: new disk, initializing [10438.164266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10438.224150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10439.573081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10439.579711] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10439.629056] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10441.235978] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10445.240166] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10446.923730] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing lnet_if_list [10449.400541] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [10449.414435] LNet: Added LNI 192.168.203.102@tcp42 [8/256/0/180] [10460.131819] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10460.133818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10460.136476] LustreError: Skipped 1 previous similar message [10460.137376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10460.142227] Lustre: Skipped 1 previous similar message [10460.150140] Lustre: Skipped 1 previous similar message [10464.050142] LustreError: 478565:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10464.132869] Lustre: server umount lustre-OST0000 complete [10467.296254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10467.299398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10467.305249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10470.369322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10470.370203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10470.377541] Lustre: Skipped 1 previous similar message [10475.489766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10475.492785] Lustre: Skipped 1 previous similar message [10480.095132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10480.147985] LustreError: 478767:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10480.151619] LustreError: 478767:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10480.194907] Lustre: server umount lustre-MDT0000 complete [10481.978349] LustreError: 476930:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761294786 with bad export cookie 11097752944244283437 [10481.982456] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10481.986455] LustreError: 476930:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10482.069540] LustreError: 478970:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10482.072983] LustreError: 478970:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10482.194132] Lustre: server umount lustre-MDT0001 complete [10486.958884] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [10488.501634] Key type lgssc unregistered [10488.728089] LNet: 479510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10488.733569] LNetError: 479510:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10488.736864] LNet: Removed LNI 192.168.203.102@tcp [10488.748180] LNet: Removed LNI 192.168.203.102@tcp42 [10489.133996] Key type .llcrypt unregistered [10489.135987] Key type ._llcrypt unregistered [10499.610169] Key type ._llcrypt registered [10499.611839] Key type .llcrypt registered [10499.671075] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [10500.088756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10500.141759] alg: No test for adler32 (adler32-zlib) [10501.021618] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [10501.135617] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [10502.727170] Key type lgssc registered [10503.218762] Lustre: Echo OBD driver; http://www.lustre.org/ [10506.687690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10506.697027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10507.927388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10509.564368] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10513.472757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10513.661506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10515.348531] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10518.033347] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10519.719892] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10522.896396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10523.091376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10525.771236] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10528.483166] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10543.457620] loop0: detected capacity change from 0 to 204800000 [10543.580733] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10543.589680] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10543.596509] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10543.636708] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10546.558694] loop0: detected capacity change from 0 to 204800000 [10546.644398] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10546.650529] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10546.655987] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10546.735279] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10549.202273] loop0: detected capacity change from 0 to 204800000 [10551.895346] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10551.925638] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10551.988512] Lustre: fs15246-OST0000: new disk, initializing [10551.991629] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [10552.022735] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10554.769372] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10559.873702] loop1: detected capacity change from 0 to 204800000 [10562.319974] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10562.369739] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10562.420650] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [10562.450414] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [10562.491449] Lustre: fs15246-MDT0000: new disk, initializing [10562.568960] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10562.580993] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10562.628624] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10563.682456] Lustre: 485125:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000006c418b22 x1846851461483520/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1761294883 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [10564.583629] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10572.772645] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10572.778857] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10572.795527] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10576.871490] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10576.883256] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10581.822288] LustreError: 486252:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [10581.861656] Lustre: server umount fs15246-MDT0000 complete [10581.991047] LustreError: 480727:0:(ldlm_lib.c:1245:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10581.999191] LustreError: 480727:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [10586.872567] LustreError: 486753:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [10586.880983] LustreError: 486753:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10586.905130] Lustre: server umount fs15246-OST0000 complete [10594.784939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10594.787369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10594.789856] LustreError: Skipped 1 previous similar message [10594.791194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10594.795147] Lustre: Skipped 1 previous similar message [10597.346271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10597.349193] Lustre: Skipped 2 previous similar messages [10599.484413] LustreError: 487254:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10599.492150] LustreError: 487254:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [10599.596913] Lustre: server umount lustre-OST0000 complete [10602.466634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10602.471529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10602.477594] Lustre: Skipped 1 previous similar message [10602.488257] Lustre: Skipped 1 previous similar message [10607.584694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10607.587571] Lustre: Skipped 1 previous similar message [10616.287139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10616.342281] LustreError: 487456:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10616.345031] LustreError: 487456:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10616.382202] Lustre: server umount lustre-MDT0000 complete [10617.824312] LustreError: 480731:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10617.831829] LustreError: 480731:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [10618.135239] LustreError: 480709:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761294922 with bad export cookie 10681480091948714867 [10618.139271] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10618.140389] LustreError: 480709:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10618.308170] Lustre: server umount lustre-MDT0001 complete [10622.806463] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [10624.088886] Key type lgssc unregistered [10624.252688] LNet: 488189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10624.256825] LNetError: 488189:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10624.266378] LNet: Removed LNI 192.168.203.102@tcp [10624.626162] Key type .llcrypt unregistered [10624.627667] Key type ._llcrypt unregistered [10633.814829] Key type ._llcrypt registered [10633.816323] Key type .llcrypt registered [10633.861597] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [10634.307674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10634.367313] alg: No test for adler32 (adler32-zlib) [10635.264790] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [10635.394868] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [10637.007189] Key type lgssc registered [10637.547052] Lustre: Echo OBD driver; http://www.lustre.org/ [10641.160610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10643.707994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10646.063313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10648.216691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10655.300105] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [10659.736277] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [10664.141465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10667.131321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10669.887053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10672.535981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10675.305565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10675.332824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10675.339094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10676.440358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10676.455398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10676.494417] Lustre: lustre-MDT0000: new disk, initializing [10676.529659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10676.539470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10678.106720] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10682.949426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10682.979461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10683.029230] Lustre: 492601:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10683.063239] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10683.066265] Lustre: Skipped 1 previous similar message [10683.111912] Lustre: lustre-MDT0001: new disk, initializing [10683.155612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10683.174477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10683.180170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10683.213966] Lustre: 493399:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [10683.218736] Lustre: 493399:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [10683.226479] Lustre: 493399:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [10683.231467] Lustre: 493399:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [10683.235593] Lustre: 493399:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [10683.240469] Lustre: 493399:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [10684.806478] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10688.293261] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10689.781987] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10692.630981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10692.667215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10692.790143] Lustre: lustre-OST0000: new disk, initializing [10692.792805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10692.832955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10695.448512] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10699.052576] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10700.749658] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10700.801377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10700.805565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10700.849810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10701.868624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [10703.430610] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10703.534602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10705.888600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10705.889621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10705.892485] LustreError: Skipped 1 previous similar message [10705.898463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10705.901322] Lustre: Skipped 1 previous similar message [10710.382258] LustreError: 495549:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10710.458178] Lustre: server umount lustre-OST0000 complete [10714.080402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10714.083469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10714.088338] Lustre: Skipped 1 previous similar message [10714.091226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10716.128253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10716.128707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10721.248126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10721.250694] Lustre: Skipped 2 previous similar messages [10726.367528] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10726.369301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10726.375168] Lustre: Skipped 1 previous similar message [10726.409559] LustreError: 495751:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10726.412202] LustreError: 495751:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10726.448218] Lustre: server umount lustre-MDT0000 complete [10727.982513] LustreError: 494347:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761295032 with bad export cookie 8909406813161145352 [10727.986263] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10727.988908] LustreError: 494347:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10728.060458] LustreError: 495953:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10728.063587] LustreError: 495953:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10728.166612] Lustre: server umount lustre-MDT0001 complete [10737.639793] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [10741.414503] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [10744.928981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10747.330552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10749.605658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10752.048222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10752.765764] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10611 sec ========= 04:37:35 (1761295055) [10753.433939] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:37:36 (1761295056) === [10754.771527] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:37:37 (1761295057) === [10769.120366] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [10770.173553] Key type lgssc unregistered [10770.297445] LNet: 500801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10770.300322] LNetError: 500801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10770.312483] LNet: Removed LNI 192.168.203.102@tcp [10770.604960] Key type .llcrypt unregistered [10770.606294] Key type ._llcrypt unregistered