[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 669233098 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001018] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002014] Switched APIC routing to physical x2apic. [ 0.003011] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007020] pid_max: default: 32768 minimum: 301 [ 0.008172] LSM: Security Framework initializing [ 0.009077] Yama: becoming mindful. [ 0.010045] SELinux: Initializing. [ 0.011094] *** VALIDATE selinux *** [ 0.022517] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028137] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029181] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030136] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031157] *** VALIDATE tmpfs *** [ 0.033126] *** VALIDATE proc *** [ 0.034290] *** VALIDATE cgroup *** [ 0.035014] *** VALIDATE cgroup2 *** [ 0.037096] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038170] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041017] Spectre V2 : User space: Vulnerable [ 0.042009] Speculative Store Bypass: Vulnerable [ 0.046874] debug: unmapping init [mem 0xffffffffb1a59000-0xffffffffb1a60fff] [ 0.050207] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051708] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052027] ... version: 2 [ 0.053011] ... bit width: 48 [ 0.054025] ... generic registers: 4 [ 0.055009] ... value mask: 0000ffffffffffff [ 0.056010] ... max period: 00007fffffffffff [ 0.057008] ... fixed-purpose events: 3 [ 0.058007] ... event mask: 000000070000000f [ 0.059274] rcu: Hierarchical SRCU implementation. [ 0.061523] smp: Bringing up secondary CPUs ... [ 0.062654] x86: Booting SMP configuration: [ 0.063017] .... node #0, CPUs: #1 #2 #3 [ 0.070358] smp: Brought up 1 node, 4 CPUs [ 0.072020] smpboot: Max logical packages: 1 [ 0.073009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.096026] node 0 deferred pages initialised in 21ms [ 0.102872] devtmpfs: initialized [ 0.103256] x86/mm: Memory block size: 128MB [ 0.104000] gcov: version magic: 0x41383552 [ 0.107960] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.111110] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.114326] pinctrl core: initialized pinctrl subsystem [ 0.116238] [ 0.116992] ************************************************************* [ 0.119014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.122010] ** ** [ 0.124008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.126009] ** ** [ 0.128011] ** This means that this kernel is built to expose internal ** [ 0.131007] ** IOMMU data structures, which may compromise security on ** [ 0.133007] ** your system. ** [ 0.135010] ** ** [ 0.137009] ** If you see this message and you are not debugging the ** [ 0.139012] ** kernel, report this immediately to your vendor! ** [ 0.141011] ** ** [ 0.143015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146014] ************************************************************* [ 0.149094] NET: Registered protocol family 16 [ 0.150555] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.153056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.156068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.160233] cpuidle: using governor menu [ 0.161978] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.164532] PCI: Using configuration type 1 for base access [ 0.165000] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.175009] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.177047] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.182000] cryptd: max_cpu_qlen set to 1000 [ 0.185495] ACPI: Added _OSI(Module Device) [ 0.187070] ACPI: Added _OSI(Processor Device) [ 0.188012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.190019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.194907] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.203645] ACPI: Interpreter enabled [ 0.204080] ACPI: PM: (supports S0 S3 S4 S5) [ 0.205013] ACPI: Using IOAPIC for interrupt routing [ 0.207127] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.211367] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.224391] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.229051] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.233097] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.238675] acpiphp: Slot [2] registered [ 0.240195] acpiphp: Slot [3] registered [ 0.241000] acpiphp: Slot [4] registered [ 0.241000] acpiphp: Slot [5] registered [ 0.241211] acpiphp: Slot [6] registered [ 0.243181] acpiphp: Slot [7] registered [ 0.245129] acpiphp: Slot [8] registered [ 0.246223] acpiphp: Slot [9] registered [ 0.248102] acpiphp: Slot [10] registered [ 0.250194] acpiphp: Slot [11] registered [ 0.252164] acpiphp: Slot [12] registered [ 0.254163] acpiphp: Slot [13] registered [ 0.255100] acpiphp: Slot [14] registered [ 0.257106] acpiphp: Slot [15] registered [ 0.260132] acpiphp: Slot [16] registered [ 0.261140] acpiphp: Slot [17] registered [ 0.263152] acpiphp: Slot [18] registered [ 0.265126] acpiphp: Slot [19] registered [ 0.266275] acpiphp: Slot [20] registered [ 0.268141] acpiphp: Slot [21] registered [ 0.270178] acpiphp: Slot [22] registered [ 0.272095] acpiphp: Slot [23] registered [ 0.273192] acpiphp: Slot [24] registered [ 0.275116] acpiphp: Slot [25] registered [ 0.277098] acpiphp: Slot [26] registered [ 0.279089] acpiphp: Slot [27] registered [ 0.280074] acpiphp: Slot [28] registered [ 0.282086] acpiphp: Slot [29] registered [ 0.284140] acpiphp: Slot [30] registered [ 0.287146] acpiphp: Slot [31] registered [ 0.289103] PCI host bridge to bus 0000:00 [ 0.291017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.293019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.296022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.298101] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.300019] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.303025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.305179] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.309772] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.316217] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.326013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.327747] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.337022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.343026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.349022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.354405] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.358927] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.365044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.369648] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.380016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.400008] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.410020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.419534] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.435022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.448026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.475019] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.491078] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.503019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.511024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.530016] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.542459] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.562021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.572021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.603016] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.619570] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.632016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.641014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.653018] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.665022] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.679021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.690019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.714022] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.726261] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.732019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.738024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.770021] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.793142] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.795386] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.797307] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.799316] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.801194] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.804100] iommu: Default domain type: Passthrough [ 0.807354] SCSI subsystem initialized [ 0.809126] ACPI: bus type USB registered [ 0.810090] usbcore: registered new interface driver usbfs [ 0.812084] usbcore: registered new interface driver hub [ 0.814098] usbcore: registered new device driver usb [ 0.815153] pps_core: LinuxPPS API ver. 1 registered [ 0.817009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.820068] PTP clock support registered [ 0.821136] EDAC MC: Ver: 3.0.0 [ 0.823181] PCI: Using ACPI for IRQ routing [ 0.824000] NetLabel: Initializing [ 0.824000] NetLabel: domain hash size = 128 [ 0.824009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.826091] NetLabel: unlabeled traffic allowed by default [ 0.828332] vgaarb: loaded [ 0.830141] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.832012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.841200] clocksource: Switched to clocksource kvm-clock [ 0.973576] VFS: Disk quotas dquot_6.6.0 [ 0.974941] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.977111] *** VALIDATE ramfs *** [ 0.978158] *** VALIDATE hugetlbfs *** [ 0.979545] pnp: PnP ACPI init [ 0.981869] pnp: PnP ACPI: found 6 devices [ 0.999039] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.001852] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.003655] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.005294] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.007372] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.009372] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.011706] NET: Registered protocol family 2 [ 1.013964] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.018228] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.021759] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.026611] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.029734] TCP: Hash tables configured (established 65536 bind 65536) [ 1.032422] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.035834] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.039467] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.042514] NET: Registered protocol family 1 [ 1.047375] RPC: Registered named UNIX socket transport module. [ 1.050214] RPC: Registered udp transport module. [ 1.051915] RPC: Registered tcp transport module. [ 1.053530] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.056247] NET: Registered protocol family 44 [ 1.057721] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.060438] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.062241] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.064531] PCI: CLS 0 bytes, default 64 [ 1.065960] Unpacking initramfs... [ 3.024674] debug: unmapping init [mem 0xffff9877fcc54000-0xffff9877fffbffff] [ 3.029835] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.032550] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.035701] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.747513] Initialise system trusted keyrings [ 3.749530] Key type blacklist registered [ 3.752443] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.762667] zbud: loaded [ 3.766134] *** VALIDATE nfs *** [ 3.767426] *** VALIDATE nfs4 *** [ 3.769092] pstore: using deflate compression [ 3.775848] Platform Keyring initialized [ 3.964195] NET: Registered protocol family 38 [ 3.965816] Key type asymmetric registered [ 3.967958] Asymmetric key parser 'x509' registered [ 3.969986] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.974141] io scheduler mq-deadline registered [ 3.975858] io scheduler kyber registered [ 3.977598] io scheduler bfq registered [ 3.979368] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.984285] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.991710] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.994899] ACPI: Power Button [PWRF] [ 4.129760] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.228802] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.442851] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.540563] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.783583] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.824033] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.873455] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.883467] Non-volatile memory driver v1.3 [ 4.885190] Linux agpgart interface v0.103 [ 4.931548] virtio_blk virtio1: [vda] 134040 512-byte logical blocks (68.6 MB/65.4 MiB) [ 4.937509] vda: detected capacity change from 0 to 68628480 [ 4.962187] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.965737] vdb: detected capacity change from 0 to 1073741824 [ 4.985525] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.989509] vdc: detected capacity change from 0 to 2621440000 [ 5.008208] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.011363] vdd: detected capacity change from 0 to 2621440000 [ 5.033762] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.039163] vde: detected capacity change from 0 to 4294967296 [ 5.058993] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.065728] vdf: detected capacity change from 0 to 4294967296 [ 5.076696] libphy: Fixed MDIO Bus: probed [ 5.087194] usbcore: registered new interface driver usbserial_generic [ 5.090485] usbserial: USB Serial support registered for generic [ 5.092690] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.103714] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.107555] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.112535] mousedev: PS/2 mouse device common for all mice [ 5.117535] rtc_cmos 00:05: RTC can wake from S4 [ 5.123719] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.125222] rtc_cmos 00:05: registered as rtc0 [ 5.130895] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.134424] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.137809] intel_pstate: CPU model not supported [ 5.150408] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.157827] hid: raw HID events driver (C) Jiri Kosina [ 5.158848] usbcore: registered new interface driver usbhid [ 5.158854] usbhid: USB HID core driver [ 5.158985] drop_monitor: Initializing network drop monitor service [ 5.159134] Initializing XFRM netlink socket [ 5.159502] NET: Registered protocol family 10 [ 5.164664] Segment Routing with IPv6 [ 5.178829] NET: Registered protocol family 17 [ 5.181808] mpls_gso: MPLS GSO support [ 5.187775] RAS: Correctable Errors collector initialized. [ 5.190795] AVX version of gcm_enc/dec engaged. [ 5.192573] AES CTR mode by8 optimization enabled [ 5.449443] sched_clock: Marking stable (5449315931, 0)->(6958083390, -1508767459) [ 5.459383] registered taskstats version 1 [ 5.466613] Loading compiled-in X.509 certificates [ 5.470464] zswap: loaded using pool lzo/zbud [ 5.547933] Key type big_key registered [ 5.584956] Key type encrypted registered [ 5.588467] ima: No TPM chip found, activating TPM-bypass! [ 5.591548] ima: Allocated hash algorithm: sha1 [ 5.593591] ima: No architecture policies found [ 5.595564] evm: Initialising EVM extended attributes: [ 5.597654] evm: security.selinux [ 5.598961] evm: security.ima [ 5.600502] evm: security.capability [ 5.602175] evm: HMAC attrs: 0x1 [ 5.605828] rtc_cmos 00:05: setting system clock to 2025-12-29 18:54:59 UTC (1767034499) [ 5.616801] debug: unmapping init [mem 0xffffffffb2a03000-0xffffffffb2bfffff] [ 5.623633] debug: unmapping init [mem 0xffffffffb1782000-0xffffffffb1a58fff] [ 5.632087] Write protecting the kernel read-only data: 28672k [ 5.638524] debug: unmapping init [mem 0xffffffffafe03000-0xffffffffafffffff] [ 5.645112] debug: unmapping init [mem 0xffffffffb0714000-0xffffffffb07fffff] [ 5.713230] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.725530] systemd[1]: Detected virtualization kvm. [ 5.729358] systemd[1]: Detected architecture x86-64. [ 5.735631] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.758557] systemd[1]: No hostname configured. [ 5.763860] systemd[1]: Set hostname to . [ 5.765869] random: systemd: uninitialized urandom read (16 bytes read) [ 5.768278] systemd[1]: Initializing machine ID from random generator. [ 6.019565] random: systemd: uninitialized urandom read (16 bytes read) [ 6.021961] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 6.026417] random: systemd: uninitialized urandom read (16 bytes read) [ 6.031330] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 6.052321] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.348993] device-mapper: uevent: version 1.0.3 [ 7.351866] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.580706] virtio_net virtio0 ens2: renamed from eth0 [ 8.651155] random: fast init done [ 9.272403] scsi host0: ata_piix [ 9.531431] scsi host1: ata_piix [ 9.544580] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.546820] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.119288] random: crng init done [ 14.120647] random: 7 urandom warning(s) missed due to ratelimiting [ 17.082727] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 17.835779] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.570686] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.026693] SELinux: Disabled at runtime. [ 20.139113] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.157822] systemd[1]: Detected virtualization kvm. [ 20.159601] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.622336] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.635174] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.659741] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.680533] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.691326] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.726509] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.761667] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ 22.273726] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 22.942196] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.033097] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.055789] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.336345] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.442178] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 29.082170] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 29.975893] NFS: Registering the id_resolver key type [ 29.977436] Key type id_resolver registered [ 29.978648] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg405-server login: [ 67.111693] libcfs: loading out-of-tree module taints kernel. [ 67.129271] Key type ._llcrypt registered [ 67.130658] Key type .llcrypt registered [ 67.169128] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 74.561698] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 75.117665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.124359] alg: No test for adler32 (adler32-zlib) [ 76.119652] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 76.436799] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 78.066044] Key type lgssc registered [ 78.781312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.948647] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 86.404647] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 89.760131] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.817559] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 96.082551] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 102.422310] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 107.448306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.465660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 107.472455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 108.560694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 108.574037] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 108.613709] Lustre: lustre-MDT0000: new disk, initializing [ 108.644697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 108.653921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 110.118410] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 117.464636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 117.516748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 117.558726] Lustre: 6488:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 117.589528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 117.593742] Lustre: Skipped 1 previous similar message [ 117.677710] Lustre: lustre-MDT0001: new disk, initializing [ 117.737658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 117.755079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 117.758687] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 120.568927] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 124.851480] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 130.810404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 130.864991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 131.011468] Lustre: lustre-OST0000: new disk, initializing [ 131.017360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 131.055637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 133.723912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 133.733824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 133.772883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 134.966425] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 146.649945] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 146.695835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 146.753760] Lustre: lustre-OST0001: new disk, initializing [ 146.758695] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 146.800952] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 149.620104] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.593338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 153.597501] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 153.622615] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 156.154833] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 163.161715] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 171.149981] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing check_logdir /tmp/testlogs/ [ 173.155964] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing yml_node [ 175.384122] Lustre: DEBUG MARKER: Client: 2.17.0.RC3 [ 176.413193] Lustre: DEBUG MARKER: MDS: 2.17.0.RC3 [ 177.447426] Lustre: DEBUG MARKER: OSS: 2.17.0.RC3 [ 178.114620] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Dec 29 13:57:51 EST 2025 [ 186.910803] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 187.626914] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 204.768735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 204.770072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 204.772654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 204.781862] Lustre: Skipped 1 previous similar message [ 206.159952] LustreError: 12188:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 206.199928] Lustre: server umount lustre-MDT0000 complete [ 208.316238] LustreError: 6481:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767034702 with bad export cookie 4251639119656408491 [ 208.318077] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 208.323217] LustreError: 6481:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 209.888888] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 209.894519] Lustre: Skipped 2 previous similar messages [ 209.898015] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 209.901024] Lustre: Skipped 3 previous similar messages [ 213.923071] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 213.928116] Lustre: Skipped 1 previous similar message [ 214.502912] LustreError: 12391:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 214.507799] LustreError: 12391:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 214.602772] Lustre: server umount lustre-MDT0001 complete [ 216.970087] LustreError: 12592:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 216.972238] LustreError: 12592:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 217.002513] Lustre: server umount lustre-OST0000 complete [ 219.118168] LustreError: 12792:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 219.121319] LustreError: 12792:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 219.201300] Lustre: server umount lustre-OST0001 complete [ 222.601734] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 227.194362] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 232.624076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 236.179809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 239.630583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 242.843768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 246.479495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 246.513881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 246.622722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 246.641730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 246.685775] Lustre: lustre-MDT0000: new disk, initializing [ 246.724586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 246.734417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 248.420921] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 254.029649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 254.057777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 254.107041] Lustre: 14972:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 254.290500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 256.326762] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 260.471352] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 262.139617] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 265.249695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 265.285526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 265.419037] Lustre: lustre-OST0000: new disk, initializing [ 265.421103] Lustre: Skipped 1 previous similar message [ 265.424035] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 265.427954] Lustre: Skipped 2 previous similar messages [ 265.470032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 265.473081] Lustre: Skipped 1 previous similar message [ 266.508437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 266.512794] Lustre: Skipped 1 previous similar message [ 266.515216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 266.566470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 268.035447] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 271.853813] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 273.599220] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 273.702190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 275.407933] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 275.503682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 276.965794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 276.966209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 276.966741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 276.972364] Lustre: Skipped 2 previous similar messages [ 276.975422] LustreError: Skipped 1 previous similar message [ 282.083781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 282.087473] Lustre: Skipped 2 previous similar messages [ 282.419265] LustreError: 17911:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 282.425587] LustreError: 17911:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 282.482897] Lustre: server umount lustre-OST0000 complete [ 285.154516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 285.159680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.463815] LustreError: 18112:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 290.467726] LustreError: 18112:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 290.525708] Lustre: server umount lustre-MDT0000 complete [ 292.324920] LustreError: 14978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 292.339613] LustreError: 14978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 292.555195] LustreError: 15752:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767034786 with bad export cookie 4251639119656410024 [ 292.555463] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 292.564101] LustreError: 15752:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 292.792369] Lustre: server umount lustre-MDT0001 complete [ 295.007414] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 13:59:47 (1767034787) [ 296.316613] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 297.194356] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 13:59:49 (1767034789) [ 300.384519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 300.664112] LustreError: 18983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 300.723956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 302.905679] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 306.146908] LustreError: 18983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.752809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 310.238684] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 313.170254] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 315.156795] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 318.827234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 322.038432] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 325.090954] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 340.961251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 340.965426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.970380] Lustre: Skipped 1 previous similar message [ 340.972832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 340.975488] Lustre: Skipped 3 previous similar messages [ 344.701082] LustreError: 21249:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 344.709546] LustreError: 21249:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 344.773723] Lustre: server umount lustre-OST0000 complete [ 353.038581] Lustre: server umount lustre-MDT0000 complete [ 353.250551] LustreError: 18978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 353.260153] LustreError: 18978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 355.305443] LustreError: 18961:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767034849 with bad export cookie 4251639119656410906 [ 355.310669] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 355.311971] LustreError: 18961:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 355.538038] Lustre: server umount lustre-MDT0001 complete [ 361.779465] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 363.719158] Key type lgssc unregistered [ 363.938164] LNet: 22185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.943478] LNetError: 22185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 363.959130] LNet: Removed LNI 192.168.204.105@tcp [ 364.498402] Key type .llcrypt unregistered [ 364.499973] Key type ._llcrypt unregistered [ 377.993483] Key type ._llcrypt registered [ 377.994962] Key type .llcrypt registered [ 378.067853] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 378.698902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 378.707082] alg: No test for adler32 (adler32-zlib) [ 379.687996] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 379.831449] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 381.434194] Key type lgssc registered [ 382.187588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 387.059498] loop: module loaded [ 387.103490] loop0: detected capacity change from 0 to 204800000 [ 387.266448] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 387.273044] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 387.437229] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 387.482334] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 390.699468] loop0: detected capacity change from 0 to 204800000 [ 390.810771] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 390.823145] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 390.831386] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 390.913630] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 394.021748] loop0: detected capacity change from 0 to 204800000 [ 397.645270] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 397.685858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 397.699516] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.886755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 398.928697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 398.977091] Lustre: lustre-MDT0000: new disk, initializing [ 399.065551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 399.085115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 401.673778] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 407.742807] loop1: detected capacity change from 0 to 204800000 [ 411.191889] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 411.236222] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 411.378807] Lustre: lustre-OST07c6: new disk, initializing [ 411.381837] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 411.384186] Lustre: Skipped 1 previous similar message [ 411.431823] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 414.777128] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 416.814534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 416.820183] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 416.849674] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 417.999911] Lustre: 24338:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 249 < left 264, rollback = 2 [ 418.004612] Lustre: 24338:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 418.007842] Lustre: 24338:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 418.011726] Lustre: 24338:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 418.016513] Lustre: 24338:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 4/65/5, delete: 0/0/0 [ 418.020938] Lustre: 24338:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 434.638276] Lustre: 25834:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 434.642959] Lustre: 25834:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 434.650841] Lustre: 25834:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 434.655663] Lustre: 25834:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 434.663907] Lustre: 25834:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 434.666868] Lustre: 25834:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 438.835184] Lustre: 24338:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 438.841116] Lustre: 24338:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 438.845444] Lustre: 24338:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 438.850189] Lustre: 24338:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 438.854649] Lustre: 24338:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 438.862728] Lustre: 24338:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 438.866571] Lustre: 24338:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 438.876112] Lustre: 24338:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 438.880848] Lustre: 24338:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 438.885557] Lustre: 24338:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 438.889986] Lustre: 24338:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 438.894887] Lustre: 24338:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 442.040472] Lustre: 24338:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 442.046689] Lustre: 24338:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 442.054714] Lustre: 24338:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 442.061130] Lustre: 24338:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 442.066088] Lustre: 24338:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 442.071623] Lustre: 24338:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 442.074749] Lustre: 24338:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 442.078806] Lustre: 24338:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 442.085553] Lustre: 24338:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 442.090839] Lustre: 24338:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 442.093900] Lustre: 24338:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 442.099037] Lustre: 24338:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 446.288781] Lustre: 24338:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 446.294849] Lustre: 24338:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 446.299990] Lustre: 24338:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 446.302712] Lustre: 24338:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 446.307780] Lustre: 24338:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 446.312586] Lustre: 24338:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 446.317415] Lustre: 24338:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 446.322690] Lustre: 24338:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 446.326800] Lustre: 24338:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 446.330491] Lustre: 24338:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 446.335024] Lustre: 24338:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 446.339545] Lustre: 24338:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 456.662792] Lustre: 24337:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 456.666790] Lustre: 24337:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 5 previous similar messages [ 456.671946] Lustre: 24337:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 456.675576] Lustre: 24337:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 456.684223] Lustre: 24337:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 456.689238] Lustre: 24337:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 456.694546] Lustre: 24337:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 456.700354] Lustre: 24337:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 456.705201] Lustre: 24337:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 456.710580] Lustre: 24337:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 456.715518] Lustre: 24337:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 456.721519] Lustre: 24337:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 503.778831] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 503.779627] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.790738] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 508.899552] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 514.020406] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 516.064157] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 516.096154] LustreError: 26525:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 516.197860] Lustre: server umount lustre-OST07c6 complete [ 521.337138] LustreError: 27026:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 521.339897] LustreError: 27026:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 521.434653] Lustre: server umount lustre-MDT0000 complete [ 530.348575] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 532.049967] Key type lgssc unregistered [ 532.246985] LNet: 27859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 532.251393] LNetError: 27859:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 532.261221] LNet: Removed LNI 192.168.204.105@tcp [ 532.728251] Key type .llcrypt unregistered [ 532.729953] Key type ._llcrypt unregistered [ 542.799918] Key type ._llcrypt registered [ 542.801744] Key type .llcrypt registered [ 542.875019] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 543.244641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 543.305034] alg: No test for adler32 (adler32-zlib) [ 544.225414] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 544.351229] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 545.960204] Key type lgssc registered [ 546.605120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 551.125239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 551.143647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 552.451658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 554.435558] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 559.495550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 559.747293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 561.803174] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 564.839279] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 566.731540] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 570.522827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 570.757436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.525452] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 576.568704] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 577.905685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 595.427918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.429747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 595.431055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 595.433546] Lustre: Skipped 1 previous similar message [ 599.539212] LustreError: 31555:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 599.583496] Lustre: server umount lustre-OST0000 complete [ 605.665483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 605.666531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 605.670921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 605.679475] Lustre: Skipped 1 previous similar message [ 605.682432] Lustre: Skipped 1 previous similar message [ 607.368251] LustreError: 31756:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 607.370662] LustreError: 31756:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 607.414029] Lustre: server umount lustre-MDT0000 complete [ 609.124807] LustreError: 29068:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035103 with bad export cookie 10710776086298877842 [ 609.128884] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 609.130182] LustreError: 29068:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 609.205330] LustreError: 31958:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 609.208146] LustreError: 31958:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 609.283425] Lustre: server umount lustre-MDT0001 complete [ 613.869829] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 615.217962] Key type lgssc unregistered [ 615.372790] LNet: 32488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.376472] LNetError: 32488:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.382550] LNet: Removed LNI 192.168.204.105@tcp [ 615.768474] Key type .llcrypt unregistered [ 615.770095] Key type ._llcrypt unregistered [ 624.466149] Key type ._llcrypt registered [ 624.467806] Key type .llcrypt registered [ 624.511753] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 624.901960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 624.934823] alg: No test for adler32 (adler32-zlib) [ 625.823951] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 625.941958] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 627.545432] Key type lgssc registered [ 628.060065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 631.677843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 631.692575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 632.934993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 635.058619] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 639.568547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 639.781584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 641.820461] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 644.340683] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 645.927570] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 649.123382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 649.305912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 651.862099] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 654.694780] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 657.521412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 666.081198] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 666.085399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 666.090705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 668.129241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 668.130551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 668.133887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 670.383313] LustreError: 35966:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 670.430776] Lustre: server umount lustre-OST0000 complete [ 673.250032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 673.250709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 673.254883] Lustre: Skipped 1 previous similar message [ 673.260350] Lustre: Skipped 2 previous similar messages [ 678.085854] LustreError: 36167:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 678.088382] LustreError: 36167:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 678.121298] Lustre: server umount lustre-MDT0000 complete [ 678.369371] LustreError: 33718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 678.376844] LustreError: 33718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 679.945972] LustreError: 33698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035173 with bad export cookie 2961501530435066842 [ 679.951809] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 679.952746] LustreError: 33698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 680.035871] LustreError: 36369:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 680.038804] LustreError: 36369:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 680.119842] Lustre: server umount lustre-MDT0001 complete [ 684.682958] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 685.910044] Key type lgssc unregistered [ 686.055714] LNet: 36900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.059267] LNetError: 36900:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 686.070737] LNet: Removed LNI 192.168.204.105@tcp [ 686.459733] Key type .llcrypt unregistered [ 686.461211] Key type ._llcrypt unregistered [ 697.474103] Key type ._llcrypt registered [ 697.475844] Key type .llcrypt registered [ 697.522925] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 697.881977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 697.919038] alg: No test for adler32 (adler32-zlib) [ 698.798832] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 698.914548] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 700.513290] Key type lgssc registered [ 700.993583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 704.607727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 704.617754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 705.829595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 707.520472] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 711.113841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 711.268416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 712.909244] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 715.034134] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 716.563908] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 719.607096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 719.809205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 722.321027] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 724.701668] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 731.114738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 736.229569] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 736.230806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 736.238677] Lustre: Skipped 1 previous similar message [ 739.712763] LustreError: 40521:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 739.755415] Lustre: server umount lustre-MDT0000 complete [ 741.347538] LustreError: 38275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 741.354664] LustreError: 38275:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 741.379160] LustreError: 39894:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035235 with bad export cookie 7712659519635895790 [ 741.381855] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 741.390170] LustreError: 39894:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 741.499517] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [ 741.502984] Lustre: Skipped 3 previous similar messages [ 746.466401] LustreError: 39438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 746.467387] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 746.472114] LustreError: 39438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 746.475904] Lustre: Skipped 1 previous similar message [ 746.480404] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 747.521975] LustreError: 40724:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 747.524584] LustreError: 40724:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 747.582923] Lustre: server umount lustre-MDT0001 complete [ 774.112201] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 774.132282] LustreError: 40947:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 774.135818] LustreError: 40947:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 774.194665] Lustre: server umount lustre-OST0000 complete [ 782.257914] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 783.480109] Key type lgssc unregistered [ 783.640633] LNet: 41784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.654395] LNetError: 41784:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.678658] LNet: Removed LNI 192.168.204.105@tcp [ 785.053699] Key type .llcrypt unregistered [ 785.055236] Key type ._llcrypt unregistered [ 795.985471] Key type ._llcrypt registered [ 795.986965] Key type .llcrypt registered [ 796.035596] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 796.529071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 796.550179] alg: No test for adler32 (adler32-zlib) [ 797.456854] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 797.570796] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 799.176173] Key type lgssc registered [ 799.654547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 802.726143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 802.735561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 803.927993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 805.560684] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 809.074745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 809.242872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 810.874874] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 813.391062] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 814.987651] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 817.937706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 818.087461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 820.445313] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 822.848631] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 828.328976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 833.507084] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 833.510734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 833.517657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 833.520417] Lustre: Skipped 1 previous similar message [ 838.629301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 838.632294] Lustre: Skipped 2 previous similar messages [ 843.748185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 843.750342] Lustre: Skipped 2 previous similar messages [ 845.280231] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 845.309095] LustreError: 45401:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 845.353848] Lustre: server umount lustre-OST0000 complete [ 858.459280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 858.465524] Lustre: Skipped 1 previous similar message [ 858.468555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 858.471635] Lustre: Skipped 2 previous similar messages [ 863.713474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 863.716723] Lustre: Skipped 1 previous similar message [ 864.644986] LustreError: 45779:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 864.648116] LustreError: 45779:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 864.681798] Lustre: server umount lustre-MDT0000 complete [ 866.194777] LustreError: 43140:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035360 with bad export cookie 4387388068448521985 [ 866.198540] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 866.199863] LustreError: 43140:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 872.316835] LustreError: 45980:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 872.319290] LustreError: 45980:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 872.373745] Lustre: server umount lustre-MDT0001 complete [ 876.410281] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 877.502480] Key type lgssc unregistered [ 877.641514] LNet: 46513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.644893] LNetError: 46513:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.656368] LNet: Removed LNI 192.168.204.105@tcp [ 877.969138] Key type .llcrypt unregistered [ 877.970358] Key type ._llcrypt unregistered [ 887.617117] Key type ._llcrypt registered [ 887.618365] Key type .llcrypt registered [ 887.666832] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 888.035365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 888.090494] alg: No test for adler32 (adler32-zlib) [ 888.966325] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 889.062894] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 890.656228] Key type lgssc registered [ 891.116035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 894.160254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 894.167107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 895.357881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 896.904738] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 900.191507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 900.329471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 901.860514] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 903.947760] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 905.324976] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 908.007830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 908.161665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 910.489288] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 912.658646] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 920.488645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 926.438957] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 927.604207] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 938.445750] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-12-2914h10m31s [ 939.018400] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 941.026639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 941.027801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 941.031409] Lustre: Skipped 2 previous similar messages [ 941.033596] Lustre: Skipped 1 previous similar message [ 942.050276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 942.052619] Lustre: Skipped 3 previous similar messages [ 944.756089] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.5@tcp (stopping) [ 948.194463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 948.198050] Lustre: Skipped 3 previous similar messages [ 954.336103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 954.337940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 954.341488] Lustre: Skipped 3 previous similar messages [ 954.374796] LustreError: 50475:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 954.405147] Lustre: server umount lustre-MDT0000 complete [ 955.621397] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 957.035500] LustreError: 49777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 957.041098] LustreError: 49777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 957.896946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 957.938966] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 958.029169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.056598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 959.077678] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 959.387131] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 960.626524] Lustre: DEBUG MARKER: Wait for df (36046) ... [ 961.172808] Lustre: DEBUG MARKER: done [ 961.473639] LustreError: 47869:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.5@tcp arrived at 1767035455 with bad export cookie 7971797588193415716 [ 961.473639] LustreError: 47868:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.5@tcp arrived at 1767035455 with bad export cookie 7971797588193415716 [ 961.483781] LustreError: 47868:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 963.042614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 963.045198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 963.050138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 964.207098] LustreError: 51525:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 964.209647] LustreError: 51525:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 964.248248] Lustre: server umount lustre-OST0000 complete [ 969.699405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.699424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 971.709862] LustreError: 51728:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 971.711810] LustreError: 51728:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 971.736901] Lustre: server umount lustre-MDT0000 complete [ 973.017899] LustreError: 48607:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035466 with bad export cookie 7971797588193416157 [ 973.020395] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 973.021839] LustreError: 48607:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 982.499710] LustreError: 51932:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 982.502038] LustreError: 51932:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 982.549290] Lustre: server umount lustre-MDT0001 complete [ 986.371432] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 987.511813] Key type lgssc unregistered [ 987.648394] LNet: 52464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.651322] LNetError: 52464:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.669300] LNet: Removed LNI 192.168.204.105@tcp [ 988.003515] Key type .llcrypt unregistered [ 988.004914] Key type ._llcrypt unregistered [ 999.477722] Key type ._llcrypt registered [ 999.479029] Key type .llcrypt registered [ 999.517524] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 1006.035660] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1006.492984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1006.511451] alg: No test for adler32 (adler32-zlib) [ 1007.375230] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1007.466554] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1009.048215] Key type lgssc registered [ 1009.446303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1012.819772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1015.277491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1017.547033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1019.830883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1022.341633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1022.363872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1022.370576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.463330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1023.478814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1023.518576] Lustre: lustre-MDT0000: new disk, initializing [ 1023.549575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1023.558204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1025.020017] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1029.520254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1029.545940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1029.584331] Lustre: 56386:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1029.610126] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1029.612773] Lustre: Skipped 1 previous similar message [ 1029.651294] Lustre: lustre-MDT0001: new disk, initializing [ 1029.688639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1029.705337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1029.710351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1031.179291] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1034.507024] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1035.831138] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1038.461478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1038.497212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1038.598760] Lustre: lustre-OST0000: new disk, initializing [ 1038.601372] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1038.636030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1040.699521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1040.703395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1040.745574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1040.794873] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1044.151703] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1045.630303] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1045.711109] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1047.097323] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1047.186286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1051.107411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1051.107505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1051.110776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1051.116272] Lustre: Skipped 1 previous similar message [ 1056.227428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1056.229571] Lustre: Skipped 2 previous similar messages [ 1061.346896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1061.349447] Lustre: Skipped 1 previous similar message [ 1062.368229] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1062.389179] LustreError: 59328:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1062.439798] Lustre: server umount lustre-OST0000 complete [ 1065.441034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1065.444077] LustreError: Skipped 1 previous similar message [ 1065.445993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.451724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1066.466319] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1071.585244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1071.587459] Lustre: Skipped 3 previous similar messages [ 1078.240181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1078.277142] LustreError: 59530:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1078.279832] LustreError: 59530:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1078.311938] Lustre: server umount lustre-MDT0000 complete [ 1079.739223] LustreError: 57166:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035573 with bad export cookie 16445871724982470577 [ 1079.742040] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1079.744093] LustreError: 57166:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1079.818217] LustreError: 59733:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1079.820516] LustreError: 59733:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1079.883183] Lustre: server umount lustre-MDT0001 complete [ 1082.927082] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 14:12:55 (1767035575) [ 1085.509794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.712974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1087.142395] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1090.333733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1090.471705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1091.948804] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1094.041261] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1095.393672] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1097.883510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1100.094940] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1102.031848] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1115.187528] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-12-2914h13m28s [ 1115.758270] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1118.232946] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1119.115697] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1119.129968] LustreError: 60397:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1139.323651] Lustre: lustre-MDT0000: Client fc806741-19a5-44fc-93d7-894dce13144c (at 192.168.204.5@tcp) reconnecting [ 1159.176147] LustreError: 60397:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1160.005240] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1160.924332] Lustre: DEBUG MARKER: done [ 1162.720883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1162.723900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.728841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1162.731385] Lustre: Skipped 2 previous similar messages [ 1164.770858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1168.754201] LustreError: 63426:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1168.756330] LustreError: 63426:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1168.802702] Lustre: server umount lustre-OST0000 complete [ 1171.937709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1171.942881] Lustre: Skipped 2 previous similar messages [ 1176.385536] LustreError: 63627:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1176.388377] LustreError: 63627:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1176.420327] Lustre: server umount lustre-MDT0000 complete [ 1177.830263] LustreError: 60384:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035671 with bad export cookie 16445871724982471445 [ 1177.833774] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1177.834888] LustreError: 60384:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1177.963106] Lustre: server umount lustre-MDT0001 complete [ 1181.975457] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 1183.107523] Key type lgssc unregistered [ 1183.251459] LNet: 64360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.255408] LNetError: 64360:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1183.266109] LNet: Removed LNI 192.168.204.105@tcp [ 1183.606039] Key type .llcrypt unregistered [ 1183.607614] Key type ._llcrypt unregistered [ 1195.790754] Key type ._llcrypt registered [ 1195.792597] Key type .llcrypt registered [ 1195.834341] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 1202.099322] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1202.559087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1202.570833] alg: No test for adler32 (adler32-zlib) [ 1203.441899] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1203.525943] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1205.112287] Key type lgssc registered [ 1205.522501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1208.665418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1211.068769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1213.306423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1215.643739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1218.157061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1218.177859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1218.183130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1219.274341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1219.289278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1219.330218] Lustre: lustre-MDT0000: new disk, initializing [ 1219.357704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1219.365596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1220.811683] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1225.217402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1225.243087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1225.281597] Lustre: 68283:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1225.308462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1225.310946] Lustre: Skipped 1 previous similar message [ 1225.348492] Lustre: lustre-MDT0001: new disk, initializing [ 1225.384539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1225.398734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1225.402867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1226.796964] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1229.996550] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1231.250266] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1233.604803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1233.631679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1233.737113] Lustre: lustre-OST0000: new disk, initializing [ 1233.739818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1233.769566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1235.687055] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1238.885039] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1240.292915] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1242.108601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1242.113085] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1242.148328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1242.403952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 1243.841095] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1243.927846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1247.203286] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1247.203558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.206526] LustreError: Skipped 1 previous similar message [ 1247.207218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1247.210665] Lustre: Skipped 1 previous similar message [ 1250.670093] LustreError: 71236:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1250.710676] Lustre: server umount lustre-OST0000 complete [ 1252.321634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.322388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1252.326377] Lustre: Skipped 1 previous similar message [ 1252.330693] Lustre: Skipped 2 previous similar messages [ 1257.441352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1257.443185] Lustre: Skipped 1 previous similar message [ 1258.046578] LustreError: 71438:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1258.048814] LustreError: 71438:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1258.077744] Lustre: server umount lustre-MDT0000 complete [ 1259.330522] LustreError: 68318:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035753 with bad export cookie 8964890863551014448 [ 1259.335115] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1259.335491] LustreError: 68318:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1259.408512] LustreError: 71639:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1259.411232] LustreError: 71639:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1259.469863] Lustre: server umount lustre-MDT0001 complete [ 1262.339218] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 14:15:55 (1767035755) [ 1262.845439] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1263.415057] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 14:15:56 (1767035756) [ 1267.447454] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1268.883861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1269.038864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1269.373286] Lustre: Failing over lustre-MDT0000 [ 1269.374821] LustreError: 72717:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1269.378062] LustreError: 72717:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1269.380582] LustreError: 72717:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1269.445360] LustreError: 72768:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1269.448125] LustreError: 72768:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1269.489314] Lustre: server umount lustre-MDT0000 complete [ 1271.561799] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 14:16:04 (1767035764) [ 1273.803588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.959147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1275.278717] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1278.143349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1279.656502] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1281.583459] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1282.802225] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1285.021044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1285.124580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1285.126468] Lustre: Skipped 1 previous similar message [ 1287.012340] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1288.813358] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1301.622251] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1303.521593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1303.524825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1303.526894] Lustre: Skipped 2 previous similar messages [ 1303.531860] Lustre: Skipped 2 previous similar messages [ 1308.641737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1308.644361] Lustre: Skipped 2 previous similar messages [ 1309.434462] LustreError: 75704:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1309.436409] LustreError: 75704:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1309.459648] Lustre: server umount lustre-MDT0000 complete [ 1310.706099] LustreError: 73317:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035804 with bad export cookie 8964890863551015575 [ 1310.709967] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1310.710357] LustreError: 73317:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1310.822560] Lustre: server umount lustre-MDT0001 complete [ 1312.047211] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1313.632885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1316.510929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1320.928911] LustreError: 66496:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d6456a24 x1852871223074816/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1321.049759] LustreError: 74942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1321.051742] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1321.055981] LustreError: 74942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1321.093866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1321.125314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1322.458536] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1325.233283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1326.708893] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1328.435418] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1329.440367] Lustre: 66498:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767035807/real 1767035807] req@00000000f52e18ed x1852871223074432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767035823 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1329.609879] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1330.660936] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1331.488078] Lustre: 66497:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767035809/real 1767035809] req@000000006c068de9 x1852871223074560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767035825 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1333.733662] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1334.752248] Lustre: 66500:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767035812/real 1767035812] req@00000000e004422f x1852871223074688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767035828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1337.696223] Lustre: 66499:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767035814/real 1767035814] req@0000000070bdbb7d x1852871223074944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767035830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1338.850204] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1338.852879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1338.854940] Lustre: Skipped 1 previous similar message [ 1342.781034] LustreError: 78163:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1342.783710] LustreError: 78163:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1342.815062] Lustre: server umount lustre-MDT0000 complete [ 1343.969502] LustreError: 78023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1343.974961] LustreError: 78023:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1344.063416] LustreError: 74054:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035837 with bad export cookie 8964890863551017465 [ 1344.067139] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1344.067460] LustreError: 74054:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1344.173589] Lustre: server umount lustre-MDT0001 complete [ 1346.361109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1349.497234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1353.184718] LustreError: 66496:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ad6f8a8b x1852871223095040/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1353.293467] LustreError: 74942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1353.319505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.321520] Lustre: Skipped 1 previous similar message [ 1353.339721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1354.338428] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1354.342239] Lustre: Skipped 1 previous similar message [ 1354.648078] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1357.754073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1359.354744] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1361.233343] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1362.461106] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1362.920598] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1364.960185] Lustre: 66498:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767035842/real 1767035842] req@000000007cbda749 x1852871223094912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767035858 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1369.061853] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1373.153221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1373.156492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1373.162572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1373.164137] Lustre: Skipped 2 previous similar messages [ 1375.552200] LustreError: 80619:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1375.554822] LustreError: 80619:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1375.583260] Lustre: server umount lustre-MDT0000 complete [ 1376.918223] LustreError: 80533:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035870 with bad export cookie 8964890863551018837 [ 1376.921211] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1376.922284] LustreError: 80533:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1377.043266] Lustre: server umount lustre-MDT0001 complete [ 1379.346994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1380.546082] Lustre: DEBUG MARKER: files compared the same [ 1391.592444] Lustre: server umount lustre-OST0000 complete [ 1397.164888] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 1398.166388] Key type lgssc unregistered [ 1398.296409] LNet: 82249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1398.299583] LNetError: 82249:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1398.312380] LNet: Removed LNI 192.168.204.105@tcp [ 1398.610839] Key type .llcrypt unregistered [ 1398.612078] Key type ._llcrypt unregistered [ 1407.326671] Key type ._llcrypt registered [ 1407.327697] Key type .llcrypt registered [ 1407.360165] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1407.662495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1407.672252] alg: No test for adler32 (adler32-zlib) [ 1408.525113] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1408.606415] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1410.184183] Key type lgssc registered [ 1410.545996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1413.112141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1413.118531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1414.278685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1415.617128] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug +malloc all [ 1418.489161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1418.600500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1419.970704] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug +malloc all [ 1421.816699] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1423.148579] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1425.560468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1425.658076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.711965] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug +malloc all [ 1429.727714] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1439.969980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1444.321143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1444.324670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1444.329923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1445.346712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1445.346919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1445.352829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.064054] LustreError: 85887:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1446.100407] Lustre: server umount lustre-OST0000 complete [ 1449.440627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1449.443849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1449.449238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1449.451838] Lustre: Skipped 1 previous similar message [ 1449.534652] LustreError: 86090:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1449.536905] LustreError: 86090:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1449.563623] Lustre: server umount lustre-MDT0000 complete [ 1450.465082] LustreError: 84937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1450.471789] LustreError: 84937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1450.890022] LustreError: 85888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035944 with bad export cookie 13190965776202746508 [ 1450.891300] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1450.894644] LustreError: 85888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1450.959669] LustreError: 86291:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1450.962625] LustreError: 86291:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1451.016598] Lustre: server umount lustre-MDT0001 complete [ 1454.924283] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 1456.043123] Key type lgssc unregistered [ 1456.185428] LNet: 86823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1456.189474] LNetError: 86823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1456.199310] LNet: Removed LNI 192.168.204.105@tcp [ 1456.528875] Key type .llcrypt unregistered [ 1456.530422] Key type ._llcrypt unregistered [ 1466.063768] Key type ._llcrypt registered [ 1466.065293] Key type .llcrypt registered [ 1466.103059] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1466.424711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1466.464452] alg: No test for adler32 (adler32-zlib) [ 1467.314413] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1467.398757] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1468.976137] Key type lgssc registered [ 1469.331801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1471.938328] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1471.944074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1487.392454] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1492.512190] LustreError: 88169:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.105@tcp: failed processing log, type 4: rc = -110 [ 1519.136167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.025724] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1522.832704] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1525.403991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1525.438837] Lustre: *** cfs_fail_loc=706, val=0*** [ 1525.569174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1525.592186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1526.854317] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1529.708440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1529.835166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1531.144792] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1532.944134] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1534.085489] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1534.945684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1534.948183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1534.952149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1535.968840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1535.971234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1535.974455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1536.812146] LustreError: 90494:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1536.843316] Lustre: server umount lustre-OST0000 complete [ 1539.042339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1539.048876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1539.050972] Lustre: Skipped 1 previous similar message [ 1544.160958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1544.163211] Lustre: Skipped 1 previous similar message [ 1549.281339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1549.284088] Lustre: Skipped 1 previous similar message [ 1552.352119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1552.384301] LustreError: 90695:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1552.386278] LustreError: 90695:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1552.413177] Lustre: server umount lustre-MDT0000 complete [ 1553.377331] LustreError: 89145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1553.383445] LustreError: 89145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1553.791847] LustreError: 88177:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036047 with bad export cookie 11587551317060813950 [ 1553.794343] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1553.797313] LustreError: 88177:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1553.866775] LustreError: 90899:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1553.869808] LustreError: 90899:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1553.941551] Lustre: server umount lustre-MDT0001 complete [ 1557.865705] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 1558.951323] Key type lgssc unregistered [ 1559.086419] LNet: 91431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1559.090046] LNetError: 91431:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1559.097335] LNet: Removed LNI 192.168.204.105@tcp [ 1559.420058] Key type .llcrypt unregistered [ 1559.421584] Key type ._llcrypt unregistered [ 1569.055116] Key type ._llcrypt registered [ 1569.056222] Key type .llcrypt registered [ 1569.096382] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1569.459366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1569.499562] alg: No test for adler32 (adler32-zlib) [ 1570.356091] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1570.449246] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1572.032171] Key type lgssc registered [ 1572.418189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1575.244357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1575.251459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1577.642409] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1580.344038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1580.543155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1581.906668] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1584.806558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1584.936134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1586.810386] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1589.611481] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1589.618035] Lustre: MGS: Client 63a06159-9042-44a7-99e3-b245e2ad3735 (at 0@lo) reconnecting [ 1589.705497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1589.736432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1591.015033] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1599.969177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1599.972250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1599.977624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1600.994376] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1600.994927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1600.996400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1601.003319] Lustre: Skipped 1 previous similar message [ 1605.039145] LustreError: 95439:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1605.076411] Lustre: server umount lustre-OST0000 complete [ 1611.235373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1611.235407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1611.236207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1611.245132] Lustre: Skipped 1 previous similar message [ 1612.476575] LustreError: 95642:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1612.478447] LustreError: 95642:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1612.487702] Lustre: server umount lustre-MDT0000 complete [ 1613.856864] LustreError: 95844:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1613.859334] LustreError: 95844:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1613.910066] Lustre: server umount lustre-MDT0001 complete [ 1615.348281] Lustre: server umount MGS complete [ 1619.589908] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:21:52 (1767036112) [ 1629.529489] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 1632.940796] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1636.117869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1638.187774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1640.092024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1642.077883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1644.235913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1644.254853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1645.523688] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1649.531929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1649.557059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1649.642263] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1649.661444] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1649.695742] Lustre: lustre-MDT0001: new disk, initializing [ 1649.719045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1649.728413] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1649.732000] Lustre: 100554:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1649.735097] LustreError: 100554:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1651.022357] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1655.376221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1655.398433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1655.493565] Lustre: lustre-OST0000: new disk, initializing [ 1655.495474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1655.517551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.525513] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1660.449055] Lustre: 101776:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000ce9fdb56 x1852871607878912/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1767036170 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1660.764662] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1663.189000] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1663.193275] Lustre: MGS: Client afb8a7e9-3edf-4015-a3b9-ae8c1d83f306 (at 0@lo) reconnecting [ 1663.207071] Lustre: 99838:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1663.213452] LustreError: 99838:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1663.223568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1663.263745] Lustre: lustre-MDT0000: new disk, initializing [ 1663.325692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1664.352824] Lustre: 102266:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000e5e5b565 x1852871607884160/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1767036174 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1664.416499] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1664.632966] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1669.410956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1669.414804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1669.479841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1688.546675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1688.547489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1688.551224] Lustre: Skipped 1 previous similar message [ 1688.554670] Lustre: Skipped 2 previous similar messages [ 1693.665927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1693.668110] Lustre: Skipped 1 previous similar message [ 1694.319077] LustreError: 102789:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1694.321016] LustreError: 102789:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1694.361117] Lustre: server umount lustre-OST0000 complete [ 1698.786172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1698.786596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.794325] Lustre: Skipped 1 previous similar message [ 1701.823321] LustreError: 102990:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1701.825297] LustreError: 102990:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1701.835227] Lustre: server umount lustre-MDT0000 complete [ 1709.230265] Lustre: server umount lustre-MDT0001 complete [ 1710.500531] LustreError: 103395:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1710.502267] LustreError: 103395:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1710.524813] Lustre: server umount MGS complete [ 1714.411463] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:23:27 (1767036207) [ 1720.388549] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 1721.448539] Key type lgssc unregistered [ 1721.579357] LNet: 104729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1721.582436] LNetError: 104729:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1721.589436] LNet: Removed LNI 192.168.204.105@tcp [ 1721.876173] Key type .llcrypt unregistered [ 1721.877584] Key type ._llcrypt unregistered [ 1728.767947] Key type ._llcrypt registered [ 1728.768912] Key type .llcrypt registered [ 1728.806479] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1729.109784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1729.136028] alg: No test for adler32 (adler32-zlib) [ 1729.986139] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1730.059778] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1731.640167] Key type lgssc registered [ 1732.019516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1734.714843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1734.722603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1735.892436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.196325] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1740.117384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1740.246791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1741.578549] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1743.502918] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1744.811296] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1747.036254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1747.150079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1749.103437] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1750.836860] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1760.491609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1765.858459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1765.861801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1765.863221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1765.868153] Lustre: Skipped 1 previous similar message [ 1765.935466] LustreError: 108174:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1765.973605] Lustre: server umount lustre-OST0000 complete [ 1769.212426] LustreError: 108375:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1769.214446] LustreError: 108375:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1769.236513] Lustre: server umount lustre-MDT0000 complete [ 1770.476225] LustreError: 105890:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036264 with bad export cookie 5877283750170913097 [ 1770.480329] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1770.480946] LustreError: 105890:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1770.543392] LustreError: 108576:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1770.546185] LustreError: 108576:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1770.602103] Lustre: server umount lustre-MDT0001 complete [ 1774.264303] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 1775.372842] Key type lgssc unregistered [ 1775.517296] LNet: 109113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.519758] LNetError: 109113:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1775.528288] LNet: Removed LNI 192.168.204.105@tcp [ 1775.815275] Key type .llcrypt unregistered [ 1775.816192] Key type ._llcrypt unregistered [ 1782.041974] Key type ._llcrypt registered [ 1782.043439] Key type .llcrypt registered [ 1782.077358] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1782.336058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1782.364079] alg: No test for adler32 (adler32-zlib) [ 1783.236646] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1783.317929] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1784.896144] Key type lgssc registered [ 1785.277748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1787.514433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1787.518180] LustreError: 110154:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1787.522737] LustreError: 110155:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1787.525847] LustreError: 110154:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1787.527977] LustreError: 110154:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1787.531136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1787.532650] LustreError: 110154:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1788.691617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1791.229829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1791.324160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1792.596448] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1794.134739] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing lsmod [ 1795.099939] LustreError: 111330:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1795.100202] LustreError: 111331:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1795.104052] LustreError: 111330:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4998 [ 1795.107326] LustreError: 111330:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1795.111374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1795.112646] LustreError: 111330:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1795.240480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1796.577073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1796.578908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.582139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1800.673756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1800.674290] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.675851] Lustre: Skipped 1 previous similar message [ 1800.680798] Lustre: Skipped 1 previous similar message [ 1801.698049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1801.700882] Lustre: Skipped 3 previous similar messages [ 1802.236559] LustreError: 111515:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1802.260523] Lustre: server umount lustre-MDT0000 complete [ 1803.544131] LustreError: 111518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036297 with bad export cookie 10271982569381919873 [ 1803.547222] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1803.547877] LustreError: 111518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1803.720276] LustreError: 110627:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000057851294 x1852871831206784/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1803.729848] LustreError: 111721:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1803.732498] LustreError: 111721:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1803.773966] Lustre: server umount lustre-MDT0001 complete [ 1815.026391] LustreError: 111921:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1815.028134] LustreError: 111921:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1815.067398] Lustre: server umount lustre-OST0000 complete [ 1817.838247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1817.999567] LustreError: 112398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1818.007028] LustreError: 112398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1818.023400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1819.271308] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1821.903650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1823.362538] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1825.144765] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1826.361911] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1828.583177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1828.690494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1828.693106] Lustre: Skipped 1 previous similar message [ 1830.582407] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1832.388384] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1844.070149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1849.313801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1849.315058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1849.320861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1849.323311] Lustre: Skipped 1 previous similar message [ 1854.434548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1854.436456] Lustre: Skipped 1 previous similar message [ 1855.214735] LustreError: 114666:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1855.217154] LustreError: 114666:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1855.256084] Lustre: server umount lustre-OST0000 complete [ 1858.016801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1858.019802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1858.022942] Lustre: Skipped 1 previous similar message [ 1862.652974] LustreError: 114867:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1862.655573] LustreError: 114867:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1862.678930] Lustre: server umount lustre-MDT0000 complete [ 1863.893095] LustreError: 112378:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036357 with bad export cookie 10271982569381920531 [ 1863.896076] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1863.898224] LustreError: 112378:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1864.022392] Lustre: server umount lustre-MDT0001 complete [ 1867.496633] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 1868.490167] Key type lgssc unregistered [ 1868.622441] LNet: 115598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1868.625547] LNetError: 115598:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1868.636431] LNet: Removed LNI 192.168.204.105@tcp [ 1868.935148] Key type .llcrypt unregistered [ 1868.936886] Key type ._llcrypt unregistered [ 1878.378468] Key type ._llcrypt registered [ 1878.380410] Key type .llcrypt registered [ 1878.421467] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1878.768401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1878.817339] alg: No test for adler32 (adler32-zlib) [ 1879.683868] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1879.780620] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1881.368170] Key type lgssc registered [ 1881.759925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1884.707061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1884.716664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1885.879473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1887.308856] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1890.067032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1890.172659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1891.440660] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1893.191092] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1894.378330] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1896.532394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1896.637595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1898.374501] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1900.224734] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1911.019700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1917.975519] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1918.807901] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1921.509162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1921.510570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1921.511055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1921.511959] Lustre: Skipped 1 previous similar message [ 1921.517290] LustreError: Skipped 1 previous similar message [ 1925.803110] LustreError: 119343:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1925.836469] Lustre: server umount lustre-OST0000 complete [ 1931.232815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1931.235233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1931.239348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1931.240965] Lustre: Skipped 1 previous similar message [ 1933.180312] LustreError: 119548:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1933.183099] LustreError: 119548:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1933.204839] Lustre: server umount lustre-MDT0000 complete [ 1934.395146] LustreError: 116952:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036428 with bad export cookie 2581062945459203575 [ 1934.398237] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1934.399044] LustreError: 116952:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1934.456938] LustreError: 119759:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1934.458630] LustreError: 119759:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1934.497402] Lustre: server umount lustre-MDT0001 complete [ 1937.738570] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 1938.683913] Key type lgssc unregistered [ 1938.800267] LNet: 120290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.802398] LNetError: 120290:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1938.812253] LNet: Removed LNI 192.168.204.105@tcp [ 1939.074835] Key type .llcrypt unregistered [ 1939.076324] Key type ._llcrypt unregistered [ 1945.926941] Key type ._llcrypt registered [ 1945.928517] Key type .llcrypt registered [ 1945.969150] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1946.416803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1946.427215] alg: No test for adler32 (adler32-zlib) [ 1947.283687] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1947.367788] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1948.952251] Key type lgssc registered [ 1949.385457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1951.931224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1951.939900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1953.117155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1954.491134] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1957.416529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1957.525975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1958.728107] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1960.454622] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1961.613334] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1963.717590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1963.833573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1965.612584] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1967.304834] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1977.130062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1982.436095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1982.437047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1982.439018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1982.442027] Lustre: Skipped 1 previous similar message [ 1986.670142] LustreError: 123738:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1986.712196] Lustre: server umount lustre-OST0000 complete [ 1988.576789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1988.579634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.586121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1988.588687] Lustre: Skipped 1 previous similar message [ 1992.673190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1992.673743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.173699] LustreError: 123941:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1994.176466] LustreError: 123941:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1994.203159] Lustre: server umount lustre-MDT0000 complete [ 1995.537894] LustreError: 121451:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036489 with bad export cookie 789489588733966226 [ 1995.540806] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1995.542592] LustreError: 121451:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1995.605388] LustreError: 124142:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1995.607689] LustreError: 124142:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1995.660488] Lustre: server umount lustre-MDT0001 complete [ 1999.229701] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 2000.231624] Key type lgssc unregistered [ 2000.361390] LNet: 124673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2000.364501] LNetError: 124673:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2000.373337] LNet: Removed LNI 192.168.204.105@tcp [ 2000.661588] Key type .llcrypt unregistered [ 2000.662801] Key type ._llcrypt unregistered [ 2009.482978] Key type ._llcrypt registered [ 2009.484188] Key type .llcrypt registered [ 2009.519781] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2009.925394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2009.940587] alg: No test for adler32 (adler32-zlib) [ 2010.793698] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 2010.886493] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 2012.464119] Key type lgssc registered [ 2012.805994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2015.205915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2015.212215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.364729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2017.690785] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2020.484577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2020.602608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2021.852867] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2023.613796] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2024.785914] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2026.951820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2026.977754] LustreError: 127644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2026.984283] LustreError: 127644:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2027.076041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2028.882839] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2030.185347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2030.566630] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2033.885933] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2035.036865] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2035.039052] Lustre: Skipped 2 previous similar messages [ 2035.197124] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 2035.208629] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2036.360525] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2036.363309] Lustre: Skipped 2 previous similar messages [ 2041.335355] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2041.337443] Lustre: Skipped 1 previous similar message [ 2043.813457] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.5@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2043.816047] Lustre: Skipped 2 previous similar messages [ 2049.528820] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.5@tcp 192.168.0.[2,10]@tcp [ 2051.656329] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.5/24@tcp in log lustre-MDT0000 [ 2051.658977] Lustre: Skipped 2 previous similar messages [ 2056.188986] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.5/24@tcp [ 2056.191025] Lustre: Skipped 1 previous similar message [ 2061.284818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2061.285954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2061.294696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2061.296156] Lustre: Skipped 1 previous similar message [ 2064.306745] LustreError: 129946:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2064.347861] Lustre: server umount lustre-OST0000 complete [ 2066.401800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2066.402244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2066.406229] Lustre: Skipped 2 previous similar messages [ 2066.411202] Lustre: Skipped 1 previous similar message [ 2071.521285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2071.523752] Lustre: Skipped 1 previous similar message [ 2071.677420] LustreError: 130147:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2071.679260] LustreError: 130147:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2071.701620] Lustre: server umount lustre-MDT0000 complete [ 2072.952484] LustreError: 127664:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036566 with bad export cookie 11955924568656159038 [ 2072.956053] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2072.957246] LustreError: 127664:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2073.012210] LustreError: 130348:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2073.013764] LustreError: 130348:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2073.061844] Lustre: server umount lustre-MDT0001 complete [ 2076.494750] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 2077.429822] Key type lgssc unregistered [ 2077.548409] LNet: 130879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2077.551532] LNetError: 130879:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2077.560278] LNet: Removed LNI 192.168.204.105@tcp [ 2077.836347] Key type .llcrypt unregistered [ 2077.837328] Key type ._llcrypt unregistered [ 2087.795671] Key type ._llcrypt registered [ 2087.797197] Key type .llcrypt registered [ 2087.834509] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2088.182376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2088.214757] alg: No test for adler32 (adler32-zlib) [ 2089.059187] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 2089.131588] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 2090.704166] Key type lgssc registered [ 2091.052859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2093.634548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2093.641223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2094.787487] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2094.791765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2094.795792] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.5/24@tcp [ 2096.080498] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2098.874286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2098.985510] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2098.994306] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.5/24@tcp [ 2099.012548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2100.388795] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2102.212911] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2103.479037] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2105.724715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2105.839730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.667556] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2109.415396] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2119.146333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2124.258428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.258699] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2124.260568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2124.262079] Lustre: Skipped 1 previous similar message [ 2124.263958] LustreError: Skipped 1 previous similar message [ 2128.749137] LustreError: 134722:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2128.784425] Lustre: server umount lustre-OST0000 complete [ 2134.497517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2134.498169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2134.501972] Lustre: Skipped 1 previous similar message [ 2134.505599] Lustre: Skipped 2 previous similar messages [ 2136.126181] LustreError: 134923:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2136.129081] LustreError: 134923:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2136.155280] Lustre: server umount lustre-MDT0000 complete [ 2137.341973] LustreError: 132379:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036631 with bad export cookie 15101454842134913869 [ 2137.344986] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2137.346706] LustreError: 132379:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2137.414262] LustreError: 135125:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2137.415924] LustreError: 135125:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2137.463234] Lustre: server umount lustre-MDT0001 complete [ 2140.891299] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 2141.848952] Key type lgssc unregistered [ 2141.973340] LNet: 135656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2141.976542] LNetError: 135656:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2141.988295] LNet: Removed LNI 192.168.204.105@tcp [ 2142.270297] Key type .llcrypt unregistered [ 2142.271548] Key type ._llcrypt unregistered [ 2150.340595] Key type ._llcrypt registered [ 2150.341457] Key type .llcrypt registered [ 2150.374596] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 2156.317952] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2156.641427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2156.660607] alg: No test for adler32 (adler32-zlib) [ 2157.497902] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 2157.570536] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 2159.144192] Key type lgssc registered [ 2159.503700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2162.405239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2164.645460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2166.751984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2169.008232] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2171.394307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2171.417186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2171.423824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2172.519566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2172.533598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2172.571085] Lustre: lustre-MDT0000: new disk, initializing [ 2172.595710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.602572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2173.975103] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2178.292077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2178.315378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2178.353751] Lustre: 139002:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2178.378506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2178.380939] Lustre: Skipped 1 previous similar message [ 2178.415266] Lustre: lustre-MDT0001: new disk, initializing [ 2178.448358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2178.460632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2178.464722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2179.822436] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2183.074522] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2184.203137] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2186.380801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2186.400978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2186.497868] Lustre: lustre-OST0000: new disk, initializing [ 2186.499507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2186.523305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2188.337138] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2191.281326] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2192.377504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2192.382137] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2192.413711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2192.572378] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2192.640196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2193.816228] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2193.881808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2197.475646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2197.475819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2197.476633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2197.479305] LustreError: Skipped 1 previous similar message [ 2197.483497] Lustre: Skipped 1 previous similar message [ 2200.496105] LustreError: 141942:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2200.535292] Lustre: server umount lustre-OST0000 complete [ 2202.593589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2202.594239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2202.597199] Lustre: Skipped 1 previous similar message [ 2202.600143] Lustre: Skipped 1 previous similar message [ 2207.713452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2207.715393] Lustre: Skipped 1 previous similar message [ 2207.869175] LustreError: 142143:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2207.871378] LustreError: 142143:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2207.895365] Lustre: server umount lustre-MDT0000 complete [ 2209.027945] LustreError: 138994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036702 with bad export cookie 125502626205171842 [ 2209.030388] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2209.030694] LustreError: 138994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2209.089591] LustreError: 142344:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2209.092301] LustreError: 142344:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2209.139049] Lustre: server umount lustre-MDT0001 complete [ 2211.801340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2211.972019] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2213.141668] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2215.865036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2217.264242] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2218.999899] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2220.110304] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2222.308637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2222.403479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.406217] Lustre: Skipped 1 previous similar message [ 2224.158618] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2225.781152] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2227.107203] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2237.361493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 2238.590524] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2238.663712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2241.932649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2241.956128] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2242.013272] Lustre: lustre-OST0001: new disk, initializing [ 2242.015902] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2242.048168] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2243.769693] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2246.835835] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2249.734220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2249.737218] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2249.775796] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2249.975944] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2251.409969] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2251.494831] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2252.736976] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff944cc318f000.ost_server_uuid 50 [ 2253.302620] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff944cc318f000.ost_server_uuid in FULL state after 0 sec [ 2259.943169] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2259.943468] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2259.947596] LustreError: Skipped 1 previous similar message [ 2259.952309] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2259.955855] Lustre: Skipped 1 previous similar message [ 2261.423075] LustreError: 147157:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2261.424967] LustreError: 147157:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2261.441335] Lustre: server umount lustre-OST0001 complete [ 2265.061531] LustreError: 144416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2265.061761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2265.062745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2265.066251] LustreError: 144416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2265.069227] Lustre: Skipped 2 previous similar messages [ 2270.181234] LustreError: 144414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2270.185247] LustreError: 144414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2275.299771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2275.301548] LustreError: 144417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2275.302679] Lustre: Skipped 4 previous similar messages [ 2275.307056] LustreError: 144417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2277.344205] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2277.362097] LustreError: 147358:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2277.364100] LustreError: 147358:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2277.405390] Lustre: server umount lustre-OST0000 complete [ 2280.419249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2280.425463] Lustre: Skipped 1 previous similar message [ 2293.216170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2293.258862] LustreError: 147562:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2293.261499] LustreError: 147562:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2293.299262] Lustre: server umount lustre-MDT0000 complete [ 2294.567296] LustreError: 144438:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036788 with bad export cookie 125502626205172710 [ 2294.572183] LustreError: 144438:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2294.573077] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2294.714924] Lustre: server umount lustre-MDT0001 complete [ 2301.096677] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 2302.151279] Key type lgssc unregistered [ 2302.279463] LNet: 148752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2302.283370] LNetError: 148752:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2302.292508] LNet: Removed LNI 192.168.204.105@tcp [ 2302.633311] Key type .llcrypt unregistered [ 2302.634430] Key type ._llcrypt unregistered [ 2313.857213] Key type ._llcrypt registered [ 2313.859180] Key type .llcrypt registered [ 2313.898584] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 2320.149887] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2320.676855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2320.688213] alg: No test for adler32 (adler32-zlib) [ 2321.555431] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 2321.635328] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 2323.216174] Key type lgssc registered [ 2323.570979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2326.482608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2328.604153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2330.511825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2332.576781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2334.716020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2334.741021] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2334.744531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2335.822516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2335.834872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2335.875904] Lustre: lustre-MDT0000: new disk, initializing [ 2335.901460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2335.907230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2337.057679] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2340.955638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2340.979162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2341.011606] Lustre: 152672:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2341.029371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2341.030816] Lustre: Skipped 1 previous similar message [ 2341.065253] Lustre: lustre-MDT0001: new disk, initializing [ 2341.097075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2341.109847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2341.113876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2342.302732] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2345.350088] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2346.541566] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2348.642955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2348.668153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2348.753482] Lustre: lustre-OST0000: new disk, initializing [ 2348.755635] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2348.781360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2350.636576] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2353.585570] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2354.912378] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2356.218614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2356.223312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2356.251115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2357.024556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2358.427675] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2358.500632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2361.315399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2361.316258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.320595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2361.322758] Lustre: Skipped 1 previous similar message [ 2365.230135] LustreError: 155626:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2365.271766] Lustre: server umount lustre-OST0000 complete [ 2366.944911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2366.946955] LustreError: Skipped 1 previous similar message [ 2366.948156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.951918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2366.954691] Lustre: Skipped 1 previous similar message [ 2371.553385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.553590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.675410] LustreError: 155827:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2372.678239] LustreError: 155827:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2372.708308] Lustre: server umount lustre-MDT0000 complete [ 2373.968635] LustreError: 152663:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036867 with bad export cookie 4882459521260224394 [ 2373.971710] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2373.974323] LustreError: 152663:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2374.046037] LustreError: 156029:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2374.047951] LustreError: 156029:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2374.100881] Lustre: server umount lustre-MDT0001 complete [ 2376.947909] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 14:34:29 (1767036869) [ 2382.235119] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 2385.375225] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2388.807291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2391.067777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2393.125658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2395.277467] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2397.631540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2397.652939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2397.738657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2397.752580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2397.800341] Lustre: lustre-MDT0000: new disk, initializing [ 2397.826752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.833370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2399.239819] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2403.576699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2403.602084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2403.637639] Lustre: 159013:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2403.660481] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2403.663386] Lustre: Skipped 1 previous similar message [ 2403.708734] Lustre: lustre-MDT0001: new disk, initializing [ 2403.738064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2403.751171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2403.754813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2405.162622] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2408.391996] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2409.627976] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2411.900139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2411.926776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2412.026084] Lustre: lustre-OST0000: new disk, initializing [ 2412.028065] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2412.056547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.989218] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2414.840646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2414.845852] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2414.875415] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2417.163776] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2419.147898] Lustre: Failing over lustre-OST0000 [ 2419.168145] LustreError: 161458:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2419.171811] LustreError: 161458:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2419.216497] Lustre: server umount lustre-OST0000 complete [ 2432.228752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2432.342172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.347680] Lustre: Skipped 1 previous similar message [ 2432.351529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2432.358949] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2433.652787] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2434.324222] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2435.655061] Lustre: Failing over lustre-MDT0000 [ 2435.738037] LustreError: 162431:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2435.741248] LustreError: 162431:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2435.779789] Lustre: server umount lustre-MDT0000 complete [ 2436.704164] Lustre: 150888:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767036914/real 1767036914] req@000000005f0b4526 x1852872395382016/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1767036930 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2437.601741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2437.607830] Lustre: Skipped 1 previous similar message [ 2441.696105] Lustre: 150889:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767036919/real 1767036919] req@00000000c3f05646 x1852872395383168/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1767036935 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2441.707801] Lustre: 150889:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2442.720980] LustreError: 159024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2442.727565] LustreError: 159024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2443.884754] LustreError: 161371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.752203] Lustre: 150890:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767036924/real 1767036924] req@00000000d734ac2f x1852872395384320/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1767036940 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2447.841713] LustreError: 159024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.846020] LustreError: 159024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2448.844804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2448.881677] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.887185] Lustre: 159016:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.204.105@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2448.962580] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2449.001266] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2449.004337] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2450.431653] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2453.989074] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2453.997321] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2453.999602] Lustre: Skipped 1 previous similar message [ 2454.014506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2459.107821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2459.108844] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2459.111556] Lustre: Skipped 1 previous similar message [ 2459.115813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2459.119389] Lustre: Skipped 2 previous similar messages [ 2461.164070] LustreError: 163313:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2461.165949] LustreError: 163313:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2461.202734] Lustre: server umount lustre-OST0000 complete [ 2464.226244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2464.227435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2464.229568] LustreError: Skipped 1 previous similar message [ 2464.231271] Lustre: Skipped 1 previous similar message [ 2468.635927] Lustre: server umount lustre-MDT0000 complete [ 2469.345234] LustreError: 159020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.349426] LustreError: 159020:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2469.879187] LustreError: 159005:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036963 with bad export cookie 4882459521260226445 [ 2469.881704] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2469.884509] LustreError: 159005:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2470.006722] Lustre: server umount lustre-MDT0001 complete [ 2473.606967] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 2474.619324] Key type lgssc unregistered [ 2474.735295] LNet: 164249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2474.738052] LNetError: 164249:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2474.745291] LNet: Removed LNI 192.168.204.105@tcp [ 2475.066899] Key type .llcrypt unregistered [ 2475.068282] Key type ._llcrypt unregistered [ 2495.256569] Key type ._llcrypt registered [ 2495.258371] Key type .llcrypt registered [ 2495.293283] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2495.591870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2495.598971] alg: No test for adler32 (adler32-zlib) [ 2496.442656] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 2496.520509] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 2498.104180] Key type lgssc registered [ 2498.453113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2501.014712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2501.022883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2502.194087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2503.438703] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2506.465744] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2508.458786] LustreError: 165776:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2508.463938] LustreError: 165776:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2508.468310] LustreError: 165776:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2508.546554] LustreError: 166299:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2508.606569] Lustre: server umount lustre-MDT0000 complete [ 2513.671116] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 2516.955261] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2520.298615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2522.606414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2524.774367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2527.011595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2529.368667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2529.396840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2529.505165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2529.521211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2529.562538] Lustre: lustre-MDT0000: new disk, initializing [ 2529.585736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2529.594510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2530.965375] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2535.284871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2535.307573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2535.338474] Lustre: 168938:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2535.360156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2535.361882] Lustre: Skipped 1 previous similar message [ 2535.401141] Lustre: lustre-MDT0001: new disk, initializing [ 2535.436230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2535.450200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2535.454571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2536.884881] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2540.187290] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2541.466816] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2543.881845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2543.909763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2544.016689] Lustre: lustre-OST0000: new disk, initializing [ 2544.019456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2544.057823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2546.095570] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2549.245970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2549.249569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2549.258305] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2549.291356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2551.162427] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2551.789617] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:37:24 (1767037044) [ 2615.776686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.776822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2615.781282] Lustre: Skipped 1 previous similar message [ 2620.897340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2620.899417] Lustre: Skipped 4 previous similar messages [ 2621.565378] LustreError: 172440:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2621.568159] LustreError: 172440:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2621.611663] Lustre: server umount lustre-MDT0000 complete [ 2622.849445] LustreError: 168928:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037116 with bad export cookie 4902449269002414989 [ 2622.850675] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2622.853879] LustreError: 168928:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2626.016452] LustreError: 168944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.016494] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.020742] LustreError: 168944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2626.024078] Lustre: Skipped 1 previous similar message [ 2626.027930] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2627.040428] LustreError: 168944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.160398] LustreError: 168944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.160454] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2632.166419] Lustre: Skipped 1 previous similar message [ 2637.280173] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2637.280433] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2637.280438] LustreError: 168945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2637.321406] LustreError: 172643:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2637.323076] LustreError: 172643:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2637.362292] Lustre: server umount lustre-MDT0001 complete [ 2644.716408] LustreError: 172846:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2644.718390] LustreError: 172846:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2644.753049] Lustre: server umount lustre-OST0000 complete [ 2647.741171] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 2651.179407] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2654.713168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2656.900132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2658.929989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2661.153077] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2663.435840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2663.457894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2663.546625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2663.559428] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2663.598703] Lustre: lustre-MDT0000: new disk, initializing [ 2663.622692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2663.628377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2664.939321] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2669.227416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2669.253300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2669.293567] Lustre: 175179:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2669.322452] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2669.325399] Lustre: Skipped 1 previous similar message [ 2669.362659] Lustre: lustre-MDT0001: new disk, initializing [ 2669.409149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2669.413915] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2670.845399] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2674.147896] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2675.479552] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2677.899629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2677.924594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2678.026163] Lustre: lustre-OST0000: new disk, initializing [ 2678.029095] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2678.061488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2678.064736] Lustre: Skipped 1 previous similar message [ 2678.782819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2678.786241] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2678.830663] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2680.080371] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2683.240208] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2684.781557] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2684.860805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2686.276792] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2686.356422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2688.996707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2688.997358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.999432] LustreError: Skipped 1 previous similar message [ 2689.004318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2689.007916] Lustre: Skipped 1 previous similar message [ 2689.072185] LustreError: 178115:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2689.074759] LustreError: 178115:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2689.106349] Lustre: server umount lustre-OST0000 complete [ 2694.114348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.117312] Lustre: Skipped 2 previous similar messages [ 2696.481597] Lustre: server umount lustre-MDT0000 complete [ 2697.770158] LustreError: 175960:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037191 with bad export cookie 4902449269002573805 [ 2697.773612] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2697.775120] LustreError: 175960:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2697.850520] LustreError: 178518:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2697.854010] LustreError: 178518:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2697.906368] Lustre: server umount lustre-MDT0001 complete [ 2700.883495] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:39:53 (1767037193) [ 2706.064738] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 2709.370499] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2712.825578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2715.056477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2717.170094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2719.420921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2721.842577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2721.863476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2721.948798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2721.962511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2722.006541] Lustre: lustre-MDT0000: new disk, initializing [ 2722.033499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2722.041342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2723.387663] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2727.656134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2727.682573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2727.721831] Lustre: 181500:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2727.848356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2729.177177] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2732.280741] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2733.537499] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2735.876055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2735.904376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2738.002938] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2741.143568] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2741.752457] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2741.787037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2751.974052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2751.974588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2751.975487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2751.975492] Lustre: Skipped 2 previous similar messages [ 2751.977456] Lustre: Skipped 1 previous similar message [ 2751.980673] LustreError: Skipped 1 previous similar message [ 2754.479109] LustreError: 184050:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2754.481727] LustreError: 184050:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2754.527198] Lustre: server umount lustre-OST0000 complete [ 2770.400166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2770.464561] Lustre: server umount lustre-MDT0000 complete [ 2771.425415] LustreError: 182293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.430129] LustreError: 182293:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2771.843634] LustreError: 183964:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037265 with bad export cookie 4902449269002574673 [ 2771.847883] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2771.849463] LustreError: 183964:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2771.984142] Lustre: server umount lustre-MDT0001 complete [ 2775.087358] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:41:08 (1767037268) [ 2781.094037] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 2784.097849] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2787.245723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2789.338330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2791.296510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2793.151136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2795.234029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2795.251028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.327864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2795.339997] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2795.341574] Lustre: Skipped 3 previous similar messages [ 2795.375207] Lustre: lustre-MDT0000: new disk, initializing [ 2795.376382] Lustre: Skipped 2 previous similar messages [ 2795.397253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2795.398851] Lustre: Skipped 2 previous similar messages [ 2795.403620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2795.405995] Lustre: Skipped 2 previous similar messages [ 2796.648279] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2800.719505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2800.746263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2800.778178] Lustre: 187438:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2800.871570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2802.228295] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2805.240123] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2806.393448] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2808.507634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2808.536125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2808.987617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2810.526404] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2813.534079] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2819.553858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2819.555925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2819.557611] LustreError: Skipped 1 previous similar message [ 2819.558319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2819.563204] Lustre: Skipped 3 previous similar messages [ 2819.565449] Lustre: Skipped 7 previous similar messages [ 2821.486022] LustreError: 189981:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2821.488323] LustreError: 189981:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2821.531723] Lustre: server umount lustre-OST0000 complete [ 2829.025936] Lustre: server umount lustre-MDT0000 complete [ 2829.792899] LustreError: 187448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.797431] LustreError: 187448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2830.360224] LustreError: 187431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037324 with bad export cookie 4902449269002575758 [ 2830.363227] LustreError: 187431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2830.363615] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2830.479654] Lustre: server umount lustre-MDT0001 complete [ 2834.369584] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 2835.428687] Key type lgssc unregistered [ 2835.569415] LNet: 190915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2835.573321] LNetError: 190915:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2835.584320] LNet: Removed LNI 192.168.204.105@tcp [ 2835.910196] Key type .llcrypt unregistered [ 2835.911880] Key type ._llcrypt unregistered [ 2845.465937] Key type ._llcrypt registered [ 2845.467111] Key type .llcrypt registered [ 2845.506663] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2846.065749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2846.079832] alg: No test for adler32 (adler32-zlib) [ 2846.952820] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 2847.061643] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 2848.656206] Key type lgssc registered [ 2849.157573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2852.004894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2852.012625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2853.176776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2854.571658] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2857.540875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2857.676962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2859.118877] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2861.014891] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2862.333622] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2864.687385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2864.809351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2866.747496] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2868.560457] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2878.121221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2893.797129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.797147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2893.802480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2893.805736] Lustre: Skipped 1 previous similar message [ 2896.942155] LustreError: 194559:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2896.980586] Lustre: server umount lustre-OST0000 complete [ 2898.913746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2898.913943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2898.915019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2898.915579] LustreError: Skipped 1 previous similar message [ 2898.925619] Lustre: Skipped 2 previous similar messages [ 2904.033669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2904.035663] Lustre: Skipped 2 previous similar messages [ 2904.449451] LustreError: 194760:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2904.451331] LustreError: 194760:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2904.480990] Lustre: server umount lustre-MDT0000 complete [ 2905.789055] LustreError: 192271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037399 with bad export cookie 6500312339312784579 [ 2905.791805] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2905.793764] LustreError: 192271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2905.862583] LustreError: 194964:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2905.864544] LustreError: 194964:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2905.916103] Lustre: server umount lustre-MDT0001 complete [ 2909.801439] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 2910.805493] Key type lgssc unregistered [ 2910.926411] LNet: 195495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2910.928609] LNetError: 195495:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2910.938405] LNet: Removed LNI 192.168.204.105@tcp [ 2911.265864] Key type .llcrypt unregistered [ 2911.266993] Key type ._llcrypt unregistered [ 2920.701305] Key type ._llcrypt registered [ 2920.703038] Key type .llcrypt registered [ 2920.741560] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2921.065814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2921.107042] alg: No test for adler32 (adler32-zlib) [ 2921.961745] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 2922.041593] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 2923.624158] Key type lgssc registered [ 2924.016914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2926.687795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2926.694836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.857124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2929.249248] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2932.261213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2932.380803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2933.745835] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2935.684881] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2936.966903] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2939.416887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2939.521397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2941.295423] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2942.991894] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2952.872029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2958.306686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2958.307054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2958.308928] LustreError: Skipped 1 previous similar message [ 2958.313827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2958.316496] Lustre: Skipped 1 previous similar message [ 2961.647110] LustreError: 199134:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2961.684322] Lustre: server umount lustre-OST0000 complete [ 2963.592150] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff944cd9126000.ost_server_uuid 50 [ 2964.187284] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2974.230794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.231486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2974.234322] Lustre: Skipped 2 previous similar messages [ 2974.235899] Lustre: Skipped 1 previous similar message [ 2978.337492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2978.339452] Lustre: Skipped 2 previous similar messages [ 2980.352981] LustreError: 199484:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2980.355381] LustreError: 199484:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2980.384378] Lustre: server umount lustre-MDT0000 complete [ 2981.671299] LustreError: 196849:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037475 with bad export cookie 477345771652501134 [ 2981.673822] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2981.675137] LustreError: 196849:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2981.753650] LustreError: 199686:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2981.755739] LustreError: 199686:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2981.807149] Lustre: server umount lustre-MDT0001 complete [ 2984.709189] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:44:37 (1767037477) [ 2986.874407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2987.029180] LustreError: 200354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2987.036567] LustreError: 200354:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2987.053878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2988.414943] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2991.381817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2991.510062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2992.850224] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2994.781829] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2996.005112] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2998.295485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3000.357593] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3002.208955] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3004.630547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3004.656104] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3004.722878] Lustre: lustre-OST0001: new disk, initializing [ 3004.724705] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3004.756171] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3004.759050] Lustre: Skipped 1 previous similar message [ 3006.736081] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3007.041300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3007.046100] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3007.077298] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3009.957971] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3011.940479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3011.940718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3016.943101] LustreError: 203559:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3016.945746] LustreError: 203559:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3016.961071] Lustre: server umount lustre-OST0000 complete [ 3017.188259] LustreError: 201951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3017.193419] LustreError: 201951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3018.887284] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3018.952586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3020.283191] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3020.358351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3020.397608] LustreError: 201951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.517738] LustreError: 201952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.522688] LustreError: 201952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3032.548591] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3032.549073] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.549366] LustreError: 201950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3032.549375] LustreError: 201950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3032.551556] LustreError: Skipped 1 previous similar message [ 3032.551971] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3032.557147] Lustre: Skipped 1 previous similar message [ 3032.571810] Lustre: Skipped 3 previous similar messages [ 3036.463128] LustreError: 204259:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3036.465282] LustreError: 204259:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3036.510368] Lustre: server umount lustre-OST0001 complete [ 3042.786992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3042.787173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3042.788394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3042.791436] Lustre: Skipped 1 previous similar message [ 3044.000339] Lustre: server umount lustre-MDT0000 complete [ 3045.325419] LustreError: 200334:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037539 with bad export cookie 477345771652502191 [ 3045.331301] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3045.397168] LustreError: 204663:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3045.400719] LustreError: 204663:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3045.459227] Lustre: server umount lustre-MDT0001 complete [ 3047.430342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3048.744398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3049.994155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3051.328136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3053.633327] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:45:46 (1767037546) [ 3055.830275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.930106] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3055.941973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3056.021259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3057.404206] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3060.298864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3060.330147] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3060.342541] Lustre: 206151:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3061.783338] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3063.559553] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3064.756462] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3066.997508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3067.068229] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3068.934791] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3070.714374] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3072.942570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3072.964294] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3073.534111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3073.547073] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3074.955227] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3076.725437] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3078.629183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3078.629771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3078.635761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3078.637020] LustreError: Skipped 1 previous similar message [ 3078.642435] Lustre: Skipped 2 previous similar messages [ 3079.662489] LustreError: 209342:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3079.664522] LustreError: 209342:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3079.680739] Lustre: server umount lustre-OST0000 complete [ 3082.354941] LustreError: 208785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.358970] LustreError: 208785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3093.987946] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3093.990541] LustreError: Skipped 1 previous similar message [ 3096.401979] Lustre: server umount lustre-OST0001 complete [ 3099.105928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.109474] Lustre: Skipped 4 previous similar messages [ 3103.841496] Lustre: server umount lustre-MDT0000 complete [ 3104.226130] LustreError: 207588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.230735] LustreError: 207588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3105.117309] LustreError: 206884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037599 with bad export cookie 477345771652503353 [ 3105.121195] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3105.122535] LustreError: 206884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3105.252747] Lustre: server umount lustre-MDT0001 complete [ 3107.103585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3108.443394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3109.727210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3111.069359] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3113.389642] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:46:46 (1767037606) [ 3118.565707] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 3121.585512] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3124.990167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3127.168300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3129.280955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3131.370126] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3133.701144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3133.726358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3133.812947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3133.825403] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3133.867291] Lustre: lustre-MDT0000: new disk, initializing [ 3133.896709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3133.900015] Lustre: Skipped 3 previous similar messages [ 3133.907475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3135.330106] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3139.538416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3139.562164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3139.592425] Lustre: 213766:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3139.612728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3139.614601] Lustre: Skipped 1 previous similar message [ 3139.651944] Lustre: lustre-MDT0001: new disk, initializing [ 3139.696572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3139.701198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3140.978920] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3144.093569] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3145.298918] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3147.476911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3147.498718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3147.585883] Lustre: lustre-OST0000: new disk, initializing [ 3147.587539] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3148.793887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3148.799512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3148.833975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3149.497245] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3152.545936] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3153.875374] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3153.956555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3155.299048] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3155.377536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3158.062138] LustreError: 216703:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3158.063809] LustreError: 216703:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3158.106156] Lustre: server umount lustre-OST0000 complete [ 3160.032898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3160.035080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3160.038670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3160.040746] Lustre: Skipped 4 previous similar messages [ 3169.279087] Lustre: server umount lustre-MDT0000 complete [ 3170.544194] LustreError: 215511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037664 with bad export cookie 477345771652505019 [ 3170.548730] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3170.549489] LustreError: 215511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3181.068340] Lustre: server umount lustre-MDT0001 complete [ 3183.883573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3184.032509] LustreError: 217582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3185.351513] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3188.240505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3189.791377] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3191.660553] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3192.951637] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3195.333780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3197.408980] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3199.178754] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3200.665735] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3208.895567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3210.200946] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3210.273635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3214.308285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3214.310572] LustreError: Skipped 1 previous similar message [ 3216.984115] Lustre: server umount lustre-OST0000 complete [ 3219.005466] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3219.092214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3220.399881] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3244.017430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3244.143929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3244.145877] Lustre: Skipped 5 previous similar messages [ 3246.084416] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3247.813574] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3248.347445] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3254.369793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3254.371463] Lustre: Skipped 6 previous similar messages [ 3255.341051] LustreError: 222030:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3255.343378] LustreError: 222030:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3255.384296] Lustre: server umount lustre-OST0000 complete [ 3259.874737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3259.877804] Lustre: Skipped 4 previous similar messages [ 3262.750188] Lustre: server umount lustre-MDT0000 complete [ 3264.023544] LustreError: 219200:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037757 with bad export cookie 477345771652505887 [ 3264.026799] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3264.027540] LustreError: 219200:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3264.137075] Lustre: server umount lustre-MDT0001 complete [ 3267.004495] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:49:20 (1767037760) [ 3269.183670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.331437] LustreError: 223109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3270.602400] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3273.494308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3274.932551] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3276.696538] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3277.959138] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3280.267717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3282.298296] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3284.028513] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3285.415907] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3295.692728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3297.113771] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3297.199195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3299.466480] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3299.493173] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3299.554438] Lustre: lustre-OST0001: new disk, initializing [ 3299.556275] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3301.493811] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3304.617240] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3306.014403] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3308.033880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3308.036606] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3308.066151] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3308.121861] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3309.355227] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3309.420501] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3313.126521] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3313.128650] LustreError: Skipped 1 previous similar message [ 3316.161742] Lustre: server umount lustre-OST0001 complete [ 3318.047159] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3318.113636] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3319.444363] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3319.516079] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3343.168424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3345.078671] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3346.961674] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3347.541989] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3354.292641] Lustre: server umount lustre-OST0001 complete [ 3372.430748] Lustre: server umount lustre-OST0000 complete [ 3379.986846] Lustre: server umount lustre-MDT0000 complete [ 3381.210201] LustreError: 223826:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037875 with bad export cookie 477345771652506818 [ 3381.213481] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3381.215766] LustreError: 223826:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3387.326278] LustreError: 229670:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3387.328345] LustreError: 229670:0:(obd_class.h:479:obd_check_dev()) Skipped 37 previous similar messages [ 3387.375378] Lustre: server umount lustre-MDT0001 complete [ 3389.334511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3390.698127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3391.971958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3393.291963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3395.694747] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:51:28 (1767037888) [ 3398.041783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3398.145844] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3398.158132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3398.235252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3398.237449] Lustre: Skipped 5 previous similar messages [ 3399.518449] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3402.612425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3402.646856] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3402.660722] Lustre: 231160:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3404.187477] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3406.008852] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3407.194344] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3409.482092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3409.551172] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3411.561995] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3413.360165] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3417.189617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3419.132858] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3420.953849] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3421.521186] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3422.874967] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3427.043650] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3428.424634] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3428.516100] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3429.729676] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff944cc67c2800.ost_server_uuid 50 [ 3430.222659] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff944cc67c2800.ost_server_uuid in FULL state after 0 sec [ 3430.557235] Lustre: Permanently deactivating lustre-OST0001 [ 3430.563312] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3431.413239] Lustre: Permanently reactivating lustre-OST0001 [ 3436.522159] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3436.522558] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3436.523433] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3436.523438] Lustre: Skipped 19 previous similar messages [ 3436.525919] LustreError: Skipped 2 previous similar messages [ 3436.529290] Lustre: Skipped 7 previous similar messages [ 3438.533004] Lustre: server umount lustre-OST0001 complete [ 3441.635914] LustreError: 232839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3441.640680] LustreError: 232839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [ 3445.905697] Lustre: server umount lustre-OST0000 complete [ 3453.337628] Lustre: server umount lustre-MDT0000 complete [ 3454.583305] LustreError: 235584:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037948 with bad export cookie 477345771652508078 [ 3454.586677] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3454.586993] LustreError: 235584:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3454.709924] Lustre: server umount lustre-MDT0001 complete [ 3456.556784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3457.840198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3459.152377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3460.443690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3462.556426] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:52:35 (1767037955) [ 3462.901184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3465.088555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3465.189808] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3465.200355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3465.203274] Lustre: Skipped 5 previous similar messages [ 3466.595967] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3469.439725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3469.469624] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3469.471348] Lustre: Skipped 1 previous similar message [ 3469.482906] Lustre: 237330:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3470.900750] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3472.709859] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3473.927357] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3476.179579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3476.203840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3476.284433] Lustre: Permanently deactivating lustre-OST0000 [ 3476.289235] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3478.279024] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3479.727384] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3480.128787] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3482.498732] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3484.520240] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3485.218952] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3486.381266] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3486.970936] Lustre: Permanently reactivating lustre-OST0000 [ 3490.617687] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3490.641829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3503.040944] Lustre: server umount lustre-OST0001 complete [ 3510.483192] Lustre: server umount lustre-OST0000 complete [ 3526.112214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3526.184570] Lustre: server umount lustre-MDT0000 complete [ 3527.461131] LustreError: 238059:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038021 with bad export cookie 477345771652509744 [ 3527.464678] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3527.464803] LustreError: 238059:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3527.586396] Lustre: server umount lustre-MDT0001 complete [ 3531.175147] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 3532.208793] Key type lgssc unregistered [ 3532.330421] LNet: 241722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3532.332683] LNetError: 241722:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3532.342276] LNet: Removed LNI 192.168.204.105@tcp [ 3532.637675] Key type .llcrypt unregistered [ 3532.639348] Key type ._llcrypt unregistered [ 3541.669725] Key type ._llcrypt registered [ 3541.671550] Key type .llcrypt registered [ 3541.710770] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3542.008626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3542.020495] alg: No test for adler32 (adler32-zlib) [ 3542.878795] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 3542.960916] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 3544.544161] Key type lgssc registered [ 3544.936561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3546.092706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3548.411744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3548.417189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3549.589150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3550.936173] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3553.907846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3553.935278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3553.964237] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3553.976590] Lustre: 243139:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3553.980266] Lustre: Permanently deactivating lustre-MDT0001 [ 3553.985925] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3554.124103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3554.802485] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3554.806378] LustreError: 243183:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3554.810267] LustreError: 243183:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3554.813520] LustreError: 243183:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 1, failed: rc = -108 [ 3555.580484] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3557.493784] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3558.803811] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3561.201329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3561.313260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3563.282736] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3563.366086] LustreError: 245125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3563.372884] LustreError: 245125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3563.374239] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3565.112506] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3567.418341] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3567.490754] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3569.382670] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3571.196310] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3571.201528] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3574.253386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3574.253804] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3576.847673] Lustre: Permanently reactivating lustre-MDT0001 [ 3576.854142] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3576.856020] Lustre: Skipped 2 previous similar messages [ 3576.882680] Lustre: lustre-MDT0001: Client 84acfcf3-6a74-43a2-8ae7-d892d3306eb4 (at 192.168.204.5@tcp) reconnecting [ 3578.058351] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3578.591399] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3580.220699] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3580.288534] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3581.616235] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3581.700559] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3582.500656] Lustre: Permanently deactivating lustre-MDT0001 [ 3582.507929] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3582.510018] Lustre: Skipped 2 previous similar messages [ 3586.239903] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3594.722718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3594.723988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3594.726561] Lustre: Skipped 3 previous similar messages [ 3594.730053] Lustre: Skipped 3 previous similar messages [ 3598.336624] LustreError: 247215:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3598.360163] Lustre: server umount lustre-MDT0000 complete [ 3599.586245] LustreError: 247060:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038093 with bad export cookie 4096960409369973846 [ 3599.588422] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3599.589717] LustreError: 247060:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3599.663095] LustreError: 247416:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3599.665801] LustreError: 247416:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3599.709583] Lustre: server umount lustre-MDT0001 complete [ 3607.023106] LustreError: 247616:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3607.026175] LustreError: 247616:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3607.041678] Lustre: server umount lustre-OST0000 complete [ 3610.903297] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:55:03 (1767038103) [ 3614.286876] LustreError: 248622:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3614.288951] LustreError: 248622:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3614.328083] Lustre: server umount lustre-OST0001 complete [ 3616.381716] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 3619.535406] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3622.978268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3625.121667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3627.226023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3629.392260] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3631.780546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3631.804410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3631.895040] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3631.898884] Lustre: Skipped 2 previous similar messages [ 3631.908868] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3631.947836] Lustre: lustre-MDT0000: new disk, initializing [ 3631.972797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3631.981852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3633.299791] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3637.490218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3637.512553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3637.546333] Lustre: 250801:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3637.570926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3637.574255] Lustre: Skipped 1 previous similar message [ 3637.609764] Lustre: lustre-MDT0001: new disk, initializing [ 3637.651066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3637.654879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3638.969281] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3642.090987] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3643.329070] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3645.601030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3645.631599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3645.733249] Lustre: lustre-OST0000: new disk, initializing [ 3645.735447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3645.770520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3645.772504] Lustre: Skipped 1 previous similar message [ 3647.720166] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3650.815934] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3651.066232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3651.070229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3651.098757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3656.829429] LustreError: 250807:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3661.066147] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3661.091830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3661.146674] Lustre: lustre-OST0001: new disk, initializing [ 3661.149718] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3663.078513] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3666.243234] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3666.920059] LustreError: 250807:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3666.931682] LustreError: 250807:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3666.944633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3666.947563] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3666.978000] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3669.872675] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 3677.024153] LustreError: 250807:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3679.345886] Lustre: lustre-OST0001: Not available for connect from 192.168.204.5@tcp (stopping) [ 3682.275966] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3682.276468] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3682.278296] LustreError: Skipped 1 previous similar message [ 3682.282252] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3682.285433] Lustre: Skipped 1 previous similar message [ 3683.759101] LustreError: 254302:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3683.760833] LustreError: 254302:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3683.779390] Lustre: server umount lustre-OST0001 complete [ 3684.462766] LustreError: 253153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3687.396602] LustreError: 253160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3687.401146] LustreError: 253160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3692.518535] LustreError: 253153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3692.525488] LustreError: 253153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3697.121132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3697.121304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3697.123863] LustreError: Skipped 1 previous similar message [ 3697.124415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3697.129463] Lustre: Skipped 2 previous similar messages [ 3697.636572] LustreError: 253160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3697.640804] LustreError: 253160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3701.934629] LustreError: 254525:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3701.937571] LustreError: 254525:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3701.974658] Lustre: server umount lustre-OST0000 complete [ 3704.289165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3704.292438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3704.297530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3704.299137] Lustre: Skipped 1 previous similar message [ 3709.467260] Lustre: server umount lustre-MDT0000 complete [ 3710.712636] LustreError: 251594:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038204 with bad export cookie 4096960409369976667 [ 3710.717504] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3725.280245] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3725.320707] LustreError: 254928:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3725.322569] LustreError: 254928:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3725.380553] Lustre: server umount lustre-MDT0001 complete [ 3729.265982] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 3730.304913] Key type lgssc unregistered [ 3730.427299] LNet: 255461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3730.429577] LNetError: 255461:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3730.439309] LNet: Removed LNI 192.168.204.105@tcp [ 3730.749736] Key type .llcrypt unregistered [ 3730.751118] Key type ._llcrypt unregistered [ 3741.683406] Key type ._llcrypt registered [ 3741.685044] Key type .llcrypt registered [ 3741.727018] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 3748.180111] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3748.591416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3748.601958] alg: No test for adler32 (adler32-zlib) [ 3749.461510] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 3749.544737] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 3751.128133] Key type lgssc registered [ 3751.513094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3754.462962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3756.630188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3758.608699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3760.687412] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3763.107810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3763.131161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3763.137902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3764.242965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3764.254357] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3764.290676] Lustre: lustre-MDT0000: new disk, initializing [ 3764.318676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3764.327660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3765.626379] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3769.858757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3769.884633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3769.926735] Lustre: 259383:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3769.950247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3769.952493] Lustre: Skipped 1 previous similar message [ 3769.988217] Lustre: lustre-MDT0001: new disk, initializing [ 3770.023323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3770.038555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3770.042166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3771.353930] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3774.477749] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3775.682363] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3777.953184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3777.973750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3778.064103] Lustre: lustre-OST0000: new disk, initializing [ 3778.065846] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3778.095449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3779.828996] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3780.664201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3780.667021] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3780.695353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3782.928691] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3784.325995] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3784.392408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3785.685503] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3785.757640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3790.820131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3790.820399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3790.829680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3790.832572] Lustre: Skipped 1 previous similar message [ 3792.430144] LustreError: 262318:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3792.466816] Lustre: server umount lustre-OST0000 complete [ 3795.938083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3795.938842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3795.939043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3795.943815] Lustre: Skipped 2 previous similar messages [ 3795.948051] LustreError: Skipped 1 previous similar message [ 3799.807415] LustreError: 262519:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3799.809430] LustreError: 262519:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3799.836969] Lustre: server umount lustre-MDT0000 complete [ 3801.077120] LustreError: 259375:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038294 with bad export cookie 14923472885795898712 [ 3801.080926] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3801.146081] LustreError: 262720:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3801.149713] LustreError: 262720:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3801.201371] Lustre: server umount lustre-MDT0001 complete [ 3804.082368] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 14:58:17 (1767038297) [ 3806.242451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3806.398085] LustreError: 263390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3806.402877] LustreError: 263390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3806.421740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3807.704106] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3810.577274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3810.707803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3812.102060] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3813.878288] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3815.021395] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3817.271209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3819.322886] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3821.087532] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3876.832833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3876.835442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3876.839722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3876.841497] Lustre: Skipped 1 previous similar message [ 3880.621659] LustreError: 265733:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3880.624208] LustreError: 265733:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3880.669245] Lustre: server umount lustre-OST0000 complete [ 3882.102410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3886.159564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3886.268705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3886.271984] Lustre: Skipped 1 previous similar message [ 3888.127810] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3889.910020] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3897.572293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3897.576397] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3902.693173] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3902.696502] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3923.427923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.428383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3923.431108] Lustre: Skipped 1 previous similar message [ 3923.433183] LustreError: Skipped 1 previous similar message [ 3923.435380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3923.437950] Lustre: Skipped 3 previous similar messages [ 3928.547534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3928.551178] Lustre: Skipped 1 previous similar message [ 3929.259118] LustreError: 267397:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3929.260854] LustreError: 267397:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3929.304268] Lustre: server umount lustre-OST0000 complete [ 3933.665814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3933.665934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3933.672125] Lustre: Skipped 1 previous similar message [ 3936.673075] Lustre: server umount lustre-MDT0000 complete [ 3937.903381] LustreError: 267601:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038431 with bad export cookie 14923472885795899580 [ 3937.906061] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3937.906743] LustreError: 267601:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3937.978485] LustreError: 267803:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3937.980212] LustreError: 267803:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3938.038815] Lustre: server umount lustre-MDT0001 complete [ 3941.570619] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 3942.623730] Key type lgssc unregistered [ 3942.756548] LNet: 268334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3942.760265] LNetError: 268334:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3942.768388] LNet: Removed LNI 192.168.204.105@tcp [ 3943.084422] Key type .llcrypt unregistered [ 3943.086165] Key type ._llcrypt unregistered [ 3952.181055] Key type ._llcrypt registered [ 3952.182101] Key type .llcrypt registered [ 3952.228973] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3952.695156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3952.756116] alg: No test for adler32 (adler32-zlib) [ 3953.625930] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 3953.720752] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 3955.312185] Key type lgssc registered [ 3955.761243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3958.561884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3958.566880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3959.715231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3961.108352] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3963.940619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3964.055158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3965.417966] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3967.192587] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3968.369279] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3970.689882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3970.811457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3972.755610] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3974.556476] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3985.131809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3999.713049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3999.715468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3999.719380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4000.737908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4000.738822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4000.739020] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4000.744385] Lustre: Skipped 1 previous similar message [ 4003.694088] LustreError: 272757:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4003.734032] Lustre: server umount lustre-OST0000 complete [ 4005.344923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4005.346909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4005.350514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4010.977629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4010.979763] Lustre: Skipped 2 previous similar messages [ 4011.072379] LustreError: 272959:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4011.073942] LustreError: 272959:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4011.100908] Lustre: server umount lustre-MDT0000 complete [ 4012.325747] LustreError: 269688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038506 with bad export cookie 17331923410089009730 [ 4012.329308] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4012.330104] LustreError: 269688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4012.402895] LustreError: 273161:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4012.404682] LustreError: 273161:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4012.466533] Lustre: server umount lustre-MDT0001 complete [ 4015.899322] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4016.898927] Key type lgssc unregistered [ 4017.025485] LNet: 273692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4017.027813] LNetError: 273692:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4017.038423] LNet: Removed LNI 192.168.204.105@tcp [ 4017.390241] Key type .llcrypt unregistered [ 4017.392132] Key type ._llcrypt unregistered [ 4024.489302] Key type ._llcrypt registered [ 4024.490285] Key type .llcrypt registered [ 4024.529502] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4024.873339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4024.911838] alg: No test for adler32 (adler32-zlib) [ 4025.769343] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 4025.856899] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4027.440160] Key type lgssc registered [ 4027.789825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4030.432510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4030.441251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4031.601617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4032.845160] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4035.770240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4035.894783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4037.217979] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4038.927671] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4040.072863] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4042.291156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4042.413846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4044.334032] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4046.199447] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4057.767691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4066.272968] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4066.275092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4066.278794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4068.323794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4068.323971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4068.324489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4071.344080] LustreError: 277286:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4071.387123] Lustre: server umount lustre-OST0000 complete [ 4073.442097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4073.442726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4073.445341] Lustre: Skipped 1 previous similar message [ 4073.448219] Lustre: Skipped 2 previous similar messages [ 4078.561188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4078.563511] Lustre: Skipped 1 previous similar message [ 4078.785490] LustreError: 277489:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4078.787417] LustreError: 277489:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4078.811809] Lustre: server umount lustre-MDT0000 complete [ 4080.091309] LustreError: 276489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038573 with bad export cookie 16507993645336979844 [ 4080.097049] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4080.176568] LustreError: 277691:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4080.178299] LustreError: 277691:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4080.228466] Lustre: server umount lustre-MDT0001 complete [ 4083.989558] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4085.004268] Key type lgssc unregistered [ 4085.127492] LNet: 278221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4085.129904] LNetError: 278221:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4085.140307] LNet: Removed LNI 192.168.204.105@tcp [ 4085.453939] Key type .llcrypt unregistered [ 4085.455019] Key type ._llcrypt unregistered [ 4094.725375] Key type ._llcrypt registered [ 4094.726393] Key type .llcrypt registered [ 4094.762799] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4095.077772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4095.125181] alg: No test for adler32 (adler32-zlib) [ 4095.983360] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 4096.075810] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4097.664194] Key type lgssc registered [ 4098.037783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4100.730129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4100.737721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4101.893479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4103.162545] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4106.217790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4106.358764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4107.743584] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4109.579318] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4110.831354] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4113.157292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4113.262155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4115.115115] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4116.826065] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4126.567645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4136.931838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4136.934732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4136.935737] LustreError: Skipped 1 previous similar message [ 4136.943493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4136.949123] Lustre: Skipped 1 previous similar message [ 4140.847099] LustreError: 282635:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4140.889787] Lustre: server umount lustre-OST0000 complete [ 4147.171519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4147.171557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4147.172176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4147.178265] Lustre: Skipped 2 previous similar messages [ 4148.354498] LustreError: 282836:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4148.356337] LustreError: 282836:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4148.388903] Lustre: server umount lustre-MDT0000 complete [ 4149.644207] LustreError: 279577:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038643 with bad export cookie 10079704413343356969 [ 4149.647314] LustreError: 279577:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4149.647944] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4149.713546] LustreError: 283038:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4149.715316] LustreError: 283038:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4149.768313] Lustre: server umount lustre-MDT0001 complete [ 4153.161354] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4154.177530] Key type lgssc unregistered [ 4154.309435] LNet: 283569:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4154.312746] LNetError: 283569:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4154.322320] LNet: Removed LNI 192.168.204.105@tcp [ 4154.650459] Key type .llcrypt unregistered [ 4154.652440] Key type ._llcrypt unregistered [ 4161.705974] Key type ._llcrypt registered [ 4161.708146] Key type .llcrypt registered [ 4161.760631] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4162.135909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4162.173143] alg: No test for adler32 (adler32-zlib) [ 4163.032749] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 4163.110838] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4164.688193] Key type lgssc registered [ 4165.209842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4167.844971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4167.850732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4169.037987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4170.439894] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4173.707974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4173.861411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4175.201581] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4177.155788] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4178.442695] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4181.154176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4181.290073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4183.277496] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4185.112792] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4194.599051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4199.906933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4199.907390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4199.911134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4199.912964] Lustre: Skipped 1 previous similar message [ 4201.076202] LustreError: 287197:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4201.128445] Lustre: server umount lustre-OST0000 complete [ 4204.512932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4204.515152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4204.519349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4204.520990] Lustre: Skipped 1 previous similar message [ 4204.547549] LustreError: 287399:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4204.549467] LustreError: 287399:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4204.577564] Lustre: server umount lustre-MDT0000 complete [ 4205.025121] LustreError: 284776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4205.030386] LustreError: 284776:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4205.804419] LustreError: 284729:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038699 with bad export cookie 9641370155443654293 [ 4205.807909] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4205.809510] LustreError: 284729:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4205.876039] LustreError: 287601:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4205.878894] LustreError: 287601:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4205.934582] Lustre: server umount lustre-MDT0001 complete [ 4209.674475] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4210.665294] Key type lgssc unregistered [ 4210.794416] LNet: 288132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4210.796713] LNetError: 288132:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4210.805320] LNet: Removed LNI 192.168.204.105@tcp [ 4211.099804] Key type .llcrypt unregistered [ 4211.100719] Key type ._llcrypt unregistered [ 4214.779810] Key type ._llcrypt registered [ 4214.781584] Key type .llcrypt registered [ 4214.821208] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4220.412053] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 4226.630112] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4227.113210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4227.124506] alg: No test for adler32 (adler32-zlib) [ 4227.981929] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 4228.068989] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4229.648154] Key type lgssc registered [ 4229.982619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4232.848783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4234.960210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4237.056460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4239.193801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4241.453902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4241.477163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4241.482891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4242.575416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4242.586391] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4242.621629] Lustre: lustre-MDT0000: new disk, initializing [ 4242.646573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4242.652722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4243.952273] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4247.990861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4248.013301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4248.043042] Lustre: 291615:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4248.063726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4248.065271] Lustre: Skipped 1 previous similar message [ 4248.100552] Lustre: lustre-MDT0001: new disk, initializing [ 4248.130352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4248.141137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4248.144147] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4249.496974] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4252.621517] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4253.843218] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4256.042367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4256.067418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4256.168845] Lustre: lustre-OST0000: new disk, initializing [ 4256.170731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4256.197951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4257.722987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4257.726703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4257.757536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4258.056930] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4261.146841] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4262.485799] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4262.552750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4263.829380] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4263.901585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4268.002808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4268.002930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4268.006207] LustreError: Skipped 1 previous similar message [ 4268.006861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4268.009853] Lustre: Skipped 1 previous similar message [ 4268.014030] Lustre: Skipped 1 previous similar message [ 4270.574593] LustreError: 294552:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4270.610772] Lustre: server umount lustre-OST0000 complete [ 4273.121681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4273.122265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4273.125107] Lustre: Skipped 1 previous similar message [ 4273.128165] Lustre: Skipped 1 previous similar message [ 4278.013989] LustreError: 294753:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4278.016977] LustreError: 294753:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4278.046924] Lustre: server umount lustre-MDT0000 complete [ 4279.336494] LustreError: 291605:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038773 with bad export cookie 3566575400727967886 [ 4279.339956] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4279.340099] LustreError: 291605:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4279.407850] LustreError: 294955:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4279.409403] LustreError: 294955:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4279.467597] Lustre: server umount lustre-MDT0001 complete [ 4282.269806] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 15:06:15 (1767038775) [ 4284.459505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4284.624538] LustreError: 295624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4284.631297] LustreError: 295624:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4284.648144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4285.983620] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4288.807182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4288.926815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4290.270106] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4292.073449] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4293.263137] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4295.523085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4297.418463] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4299.193794] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4319.201105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4319.205387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4319.211302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4321.251683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4321.251805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4321.251918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4323.567082] LustreError: 297894:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4323.568833] LustreError: 297894:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4323.610265] Lustre: server umount lustre-OST0000 complete [ 4326.369575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4326.373933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4326.376150] Lustre: Skipped 2 previous similar messages [ 4331.068471] LustreError: 298097:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4331.070211] LustreError: 298097:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4331.097818] Lustre: server umount lustre-MDT0000 complete [ 4331.488879] LustreError: 297896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4331.495622] LustreError: 297896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4332.293303] LustreError: 295604:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038826 with bad export cookie 3566575400727968761 [ 4332.298625] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4332.299510] LustreError: 295604:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4332.420790] Lustre: server umount lustre-MDT0001 complete [ 4335.789856] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4336.807495] Key type lgssc unregistered [ 4336.930384] LNet: 298832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4336.934409] LNetError: 298832:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4336.944316] LNet: Removed LNI 192.168.204.105@tcp [ 4337.236068] Key type .llcrypt unregistered [ 4337.237101] Key type ._llcrypt unregistered [ 4345.684309] Key type ._llcrypt registered [ 4345.685327] Key type .llcrypt registered [ 4345.720951] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 4352.042502] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4352.393970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4352.438912] alg: No test for adler32 (adler32-zlib) [ 4353.288164] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 4353.375764] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4354.960196] Key type lgssc registered [ 4355.301609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4358.108450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4360.245147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4362.337384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4364.401530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4366.054775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4367.879648] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4370.153630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4370.173817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4370.178977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4371.258545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4371.269108] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4371.303466] Lustre: lustre-MDT0000: new disk, initializing [ 4371.326680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4371.332886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4372.667790] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4376.813331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4376.834943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4376.876378] Lustre: 302607:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4376.897782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4376.899401] Lustre: Skipped 1 previous similar message [ 4376.937176] Lustre: lustre-MDT0001: new disk, initializing [ 4376.974037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4376.986489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4376.991446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4378.312360] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4381.518316] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4382.811814] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4385.020903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4385.043821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4385.130228] Lustre: lustre-OST2710: new disk, initializing [ 4385.131906] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4385.158995] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4387.091575] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4387.513207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4387.519331] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4387.552857] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4390.171310] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4392.475496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4392.501599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4392.555274] Lustre: lustre-OST03e8: new disk, initializing [ 4392.557188] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4392.588209] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4394.539403] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4396.382582] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4397.567723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4397.573429] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4397.604814] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4398.962569] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4399.029955] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4400.276390] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4400.348961] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4401.583211] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4401.652794] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4402.918773] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4402.979591] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4407.782792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4407.784954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4407.786496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4407.799442] Lustre: Skipped 2 previous similar messages [ 4410.113608] LustreError: 306952:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4410.142331] Lustre: server umount lustre-MDT0000 complete [ 4411.446101] LustreError: 302599:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038905 with bad export cookie 8967633000822694779 [ 4411.449240] LustreError: 302599:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4411.450332] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4411.514046] LustreError: 307154:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4411.516210] LustreError: 307154:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4411.563377] Lustre: server umount lustre-MDT0001 complete [ 4423.091734] LustreError: 307354:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4423.093854] LustreError: 307354:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4423.111674] Lustre: server umount lustre-OST2710 complete [ 4434.613618] LustreError: 307556:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4434.615853] LustreError: 307556:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4434.672910] Lustre: server umount lustre-OST03e8 complete [ 4440.583521] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 4443.562950] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4446.751169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4448.962446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4450.964064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4453.100062] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4455.395404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4455.419568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4455.505093] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4455.515098] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4455.561754] Lustre: lustre-MDT0000: new disk, initializing [ 4455.586252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4455.592301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4456.842604] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4461.072525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4461.095053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4461.132571] Lustre: 310347:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4461.234510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4462.509486] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4465.752648] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4467.084535] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4469.372428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4469.394725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4469.494022] Lustre: lustre-OST0000: new disk, initializing [ 4469.495350] Lustre: Skipped 1 previous similar message [ 4469.497032] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4469.499098] Lustre: Skipped 2 previous similar messages [ 4469.524678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4469.527451] Lustre: Skipped 1 previous similar message [ 4471.430738] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4471.608571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4471.611410] Lustre: Skipped 1 previous similar message [ 4471.613304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4471.650230] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4474.517248] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4475.910364] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4475.983242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4477.379904] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4477.455534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4480.174153] LustreError: 313283:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4480.176122] LustreError: 313283:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4480.216722] Lustre: server umount lustre-OST0000 complete [ 4482.017460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4482.018213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4482.019036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4482.019040] Lustre: Skipped 3 previous similar messages [ 4482.026419] Lustre: Skipped 2 previous similar messages [ 4487.137320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4487.140571] Lustre: Skipped 2 previous similar messages [ 4491.236233] LustreError: 313484:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4491.238674] LustreError: 313484:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4491.264062] Lustre: server umount lustre-MDT0000 complete [ 4492.535825] LustreError: 310338:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038986 with bad export cookie 8967633000822696445 [ 4492.541696] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4492.542048] LustreError: 310338:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4503.061158] Lustre: server umount lustre-MDT0001 complete [ 4506.008241] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 15:09:58 (1767038998) [ 4506.543923] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4507.148692] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 15:10:00 (1767039000) [ 4508.207284] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4511.290765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4512.615962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4513.860270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4515.041408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4515.370400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4517.500218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4517.585757] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4517.594328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4517.650891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4518.823794] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4521.451513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4521.483922] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4521.492334] Lustre: 315811:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4522.919606] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4524.625278] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4525.819520] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4528.029411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4528.059448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4528.132053] Lustre: Denying initial registration attempt from nid 192.168.204.105@tcp, specified as failover [ 4528.134609] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4528.137603] LustreError: 317427:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 4528.139689] LustreError: 317427:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4528.141429] LustreError: 317427:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4528.164497] Lustre: server umount lustre-OST0000 complete [ 4528.165665] LustreError: 317427:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4530.658062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4530.659035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4530.664134] Lustre: Skipped 1 previous similar message [ 4530.667490] Lustre: Skipped 1 previous similar message [ 4535.777175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4535.779440] Lustre: Skipped 1 previous similar message [ 4536.447076] LustreError: 317795:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4536.448902] LustreError: 317795:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4536.471074] Lustre: server umount lustre-MDT0000 complete [ 4537.656514] LustreError: 315803:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767039031 with bad export cookie 8967633000822697313 [ 4537.659329] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4537.662264] LustreError: 315803:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4537.777346] Lustre: server umount lustre-MDT0001 complete [ 4541.581799] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4542.584509] Key type lgssc unregistered [ 4542.715345] LNet: 318528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4542.719247] LNetError: 318528:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4542.731339] LNet: Removed LNI 192.168.204.105@tcp [ 4543.037154] Key type .llcrypt unregistered [ 4543.038246] Key type ._llcrypt unregistered [ 4546.505265] Key type ._llcrypt registered [ 4546.506465] Key type .llcrypt registered [ 4546.545422] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4546.954223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4547.004242] alg: No test for adler32 (adler32-zlib) [ 4547.853625] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 4547.935071] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4549.512218] Key type lgssc registered [ 4549.883055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4552.131168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4553.411512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4554.662539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4555.894251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4556.235920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4562.435310] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4565.224948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4565.230028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4566.321607] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4566.328721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4566.384642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4567.615346] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4570.378476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4570.408656] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4570.418576] Lustre: 321260:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4570.509399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4571.743800] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4573.412712] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4574.601147] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4576.766673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4576.791047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4576.855738] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4576.923209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4578.717597] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4580.399118] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4581.062091] LustreError: 323535:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4581.092311] Lustre: server umount lustre-OST0000 complete [ 4582.370116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.371519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4582.376382] Lustre: Skipped 1 previous similar message [ 4587.489377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4587.490866] Lustre: Skipped 2 previous similar messages [ 4588.354135] LustreError: 323736:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4588.355757] LustreError: 323736:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4588.386725] Lustre: server umount lustre-MDT0000 complete [ 4589.578990] LustreError: 321251:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767039083 with bad export cookie 11648365884963422338 [ 4589.582054] LustreError: 321251:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4589.582322] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4589.646218] LustreError: 323941:0:(obd_class.h:479:obd_check_dev()) Device 7 not setup [ 4589.647912] LustreError: 323941:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4589.696777] Lustre: server umount lustre-MDT0001 complete [ 4593.045868] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4593.960101] Key type lgssc unregistered [ 4594.076373] LNet: 324472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4594.078724] LNetError: 324472:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4594.088299] LNet: Removed LNI 192.168.204.105@tcp [ 4594.351247] Key type .llcrypt unregistered [ 4594.352232] Key type ._llcrypt unregistered [ 4603.368079] Key type ._llcrypt registered [ 4603.368994] Key type .llcrypt registered [ 4603.412034] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4603.920576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4603.933763] alg: No test for adler32 (adler32-zlib) [ 4604.778208] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 4604.854301] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4606.432158] Key type lgssc registered [ 4606.773276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4609.232267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4609.237560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.386773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4611.778756] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4614.641035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4614.765377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4616.154348] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4618.005197] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4619.233592] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4621.490849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4621.607091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4623.456714] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4625.242235] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4645.346192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4645.348798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4645.352854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4646.005381] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.5@tcp (stopping) [ 4647.394350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4647.397927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4647.401510] Lustre: Skipped 1 previous similar message [ 4648.959475] LustreError: 328113:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4648.991040] Lustre: server umount lustre-MDT0000 complete [ 4650.236175] LustreError: 325827:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767039144 with bad export cookie 8996948105745628996 [ 4650.240258] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4650.241535] LustreError: 325827:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4651.116304] LustreError: 325871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4651.120824] LustreError: 325871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4651.123306] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [ 4651.125164] Lustre: Skipped 2 previous similar messages [ 4652.512820] LustreError: 325871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4652.513311] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4655.520811] LustreError: 325871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4655.521058] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4655.528502] Lustre: Skipped 1 previous similar message [ 4656.381363] LustreError: 328315:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4656.384067] LustreError: 328315:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4656.433823] Lustre: server umount lustre-MDT0001 complete [ 4658.561394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4661.299869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4676.064642] LustreError: 325156:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000090e10400 x1852874789250304/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4676.191215] LustreError: 327453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4676.199452] LustreError: 327453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4676.213996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4676.238648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4677.435799] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4680.177911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4680.315504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4681.546052] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4683.250225] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4685.419822] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4686.935915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4688.199513] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4688.769835] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4690.916174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4690.916594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4690.918978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4690.921692] Lustre: Skipped 1 previous similar message [ 4690.924272] Lustre: Skipped 1 previous similar message [ 4696.113126] LustreError: 330618:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4696.115890] LustreError: 330618:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4696.162444] Lustre: server umount lustre-OST0000 complete [ 4701.155093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4701.155311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4701.161579] Lustre: Skipped 1 previous similar message [ 4703.549672] LustreError: 330820:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4703.551884] LustreError: 330820:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4703.574458] Lustre: server umount lustre-MDT0000 complete [ 4704.844240] LustreError: 325826:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767039198 with bad export cookie 8996948105745636150 [ 4704.847220] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4704.848562] LustreError: 325826:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4704.968852] Lustre: server umount lustre-MDT0001 complete [ 4708.815821] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4709.904880] Key type lgssc unregistered [ 4710.039431] LNet: 331554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4710.043573] LNetError: 331554:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4710.056276] LNet: Removed LNI 192.168.204.105@tcp [ 4710.356888] Key type .llcrypt unregistered [ 4710.358621] Key type ._llcrypt unregistered [ 4719.814346] Key type ._llcrypt registered [ 4719.815374] Key type .llcrypt registered [ 4719.856097] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4720.178406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4720.188693] alg: No test for adler32 (adler32-zlib) [ 4721.045704] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 4721.145304] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4722.728174] Key type lgssc registered [ 4723.113037] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4725.758134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4725.764078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4726.921521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4728.224024] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4731.107321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4731.237891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4732.537865] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4734.320519] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4735.553635] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4741.601949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4741.602341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4741.602587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4741.606953] Lustre: Skipped 1 previous similar message [ 4742.625386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4742.627224] Lustre: Skipped 2 previous similar messages [ 4742.654018] LustreError: 334318:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4742.680450] Lustre: server umount lustre-MDT0000 complete [ 4743.866190] LustreError: 332907:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767039237 with bad export cookie 12473300253535517880 [ 4743.868970] LustreError: 332907:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4743.869122] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4743.930000] LustreError: 334521:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4743.931870] LustreError: 334521:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4743.973990] Lustre: server umount lustre-MDT0001 complete [ 4748.739465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4748.841728] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4748.856259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4748.925116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4750.196366] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4752.933811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4752.964909] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4752.973776] Lustre: 335289:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4753.074260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4754.385345] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4756.138567] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4757.263765] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4759.806721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4759.878880] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4759.883278] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4759.888315] LustreError: 336949:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4759.890577] LustreError: 336949:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4759.893230] LustreError: 336949:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4759.914333] Lustre: server umount lustre-OST0000 complete [ 4759.915612] LustreError: 336949:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4762.666555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4762.731190] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4762.791298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4764.541770] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4766.242334] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4768.766890] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4768.784968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4768.792837] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4768.824249] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4768.852305] Lustre: lustre-OST0001: new disk, initializing [ 4768.853958] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4770.713450] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4772.443478] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4773.412200] LustreError: 339053:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4773.413975] LustreError: 339053:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4773.431712] Lustre: server umount lustre-OST0001 complete [ 4778.480375] LustreError: 339218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4778.481909] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4778.486195] LustreError: 339218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4778.486439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4778.489497] Lustre: Skipped 1 previous similar message [ 4780.718952] LustreError: 339255:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4780.721258] LustreError: 339255:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4780.762354] Lustre: server umount lustre-OST0000 complete [ 4783.586125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4783.590333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4783.593081] Lustre: Skipped 2 previous similar messages [ 4788.159406] LustreError: 339464:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup [ 4788.161266] LustreError: 339464:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4788.189110] Lustre: server umount lustre-MDT0000 complete [ 4788.706485] LustreError: 335300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4788.711944] LustreError: 335300:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4789.460767] LustreError: 335280:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767039283 with bad export cookie 12473300253535518419 [ 4789.464821] LustreError: 335280:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4789.465139] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4789.600732] Lustre: server umount lustre-MDT0001 complete [ 4792.886791] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4793.814053] Key type lgssc unregistered [ 4793.939375] LNet: 340196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4793.942792] LNetError: 340196:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4793.953298] LNet: Removed LNI 192.168.204.105@tcp [ 4794.246826] Key type .llcrypt unregistered [ 4794.248782] Key type ._llcrypt unregistered [ 4804.744927] Key type ._llcrypt registered [ 4804.745856] Key type .llcrypt registered [ 4804.781428] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 4811.128328] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4811.543405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4811.549238] alg: No test for adler32 (adler32-zlib) [ 4812.401314] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 4812.484027] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4814.064222] Key type lgssc registered [ 4814.411578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4817.253929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4819.571076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4821.812595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4823.841423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4826.294642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4826.314586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4826.319293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4827.404646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4827.414472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4827.448372] Lustre: lustre-MDT0000: new disk, initializing [ 4827.474116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4827.480117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4828.792431] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4833.162419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4833.188987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4833.220259] Lustre: 344116:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4833.240653] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4833.242736] Lustre: Skipped 1 previous similar message [ 4833.278595] Lustre: lustre-MDT0001: new disk, initializing [ 4833.310102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4833.321926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4833.324767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4834.540837] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4837.611334] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4838.790041] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4841.078733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4841.101617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4841.206570] Lustre: lustre-OST0000: new disk, initializing [ 4841.208540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4841.236791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4843.226598] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4846.894435] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4847.609621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4847.612689] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4847.642874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4848.214334] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4848.294576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4849.652242] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4849.736097] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4852.706536] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4852.706824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4852.708986] LustreError: Skipped 1 previous similar message [ 4852.712753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4852.716105] Lustre: Skipped 1 previous similar message [ 4856.498803] LustreError: 347058:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4856.543415] Lustre: server umount lustre-OST0000 complete [ 4857.825714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4857.826350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4857.833621] Lustre: Skipped 2 previous similar messages [ 4862.945743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4862.948567] Lustre: Skipped 2 previous similar messages [ 4868.065186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4868.066828] Lustre: Skipped 1 previous similar message [ 4872.160108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4872.162161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4872.164958] Lustre: Skipped 1 previous similar message [ 4872.201534] LustreError: 347259:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4872.203282] LustreError: 347259:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4872.238415] Lustre: server umount lustre-MDT0000 complete [ 4873.461850] LustreError: 345866:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767039367 with bad export cookie 407129060576544640 [ 4873.464808] LustreError: 345866:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4873.466188] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4873.530573] LustreError: 347462:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4873.532588] LustreError: 347462:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4873.588541] Lustre: server umount lustre-MDT0001 complete [ 4876.772111] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 15:16:09 (1767039369) [ 4878.704620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4881.019631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4889.986977] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 4893.366830] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4896.529725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4899.055558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4901.551944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4903.716160] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4905.925461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4905.949590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4906.037386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4906.047503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4906.082458] Lustre: lustre-MDT0000: new disk, initializing [ 4906.108318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4906.114697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4907.508095] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4911.926324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4911.948823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4911.979234] Lustre: 351230:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4912.013761] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4912.015709] Lustre: Skipped 1 previous similar message [ 4912.050284] Lustre: lustre-MDT0001: new disk, initializing [ 4912.080378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4912.091474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4912.094269] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4913.418312] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4916.903944] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4918.480512] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4921.419845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4921.456157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4921.588042] Lustre: lustre-OST0000: new disk, initializing [ 4921.590113] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4921.626774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4923.693458] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4926.811653] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4927.991422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4927.996162] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4928.032547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4928.176745] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4928.270151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4929.562629] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4929.634818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4933.091954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4933.092079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4933.092651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4933.096756] Lustre: Skipped 1 previous similar message [ 4936.367425] LustreError: 354167:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4936.371475] LustreError: 354167:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4936.421894] Lustre: server umount lustre-OST0000 complete [ 4937.696912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4937.699093] LustreError: Skipped 1 previous similar message [ 4937.700295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4943.745539] LustreError: 354368:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4943.747513] LustreError: 354368:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4943.778408] Lustre: server umount lustre-MDT0000 complete [ 4945.028353] LustreError: 351222:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767039438 with bad export cookie 407129060576545508 [ 4945.032063] LustreError: 351222:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4945.032729] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4945.163204] Lustre: server umount lustre-MDT0001 complete [ 4948.235650] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:17:21 (1767039441) [ 4950.965835] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:17:23 (1767039443) [ 4954.329863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4954.547801] LustreError: 355583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4954.552701] LustreError: 355583:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4954.585578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4956.011959] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4958.863820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4960.302518] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4962.069385] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4963.242591] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4965.556684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4967.502827] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4969.335530] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4988.237741] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4988.767364] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4989.361853] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4989.953994] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4990.630129] Lustre: Failing over lustre-MDT0000 [ 4990.827636] LustreError: 358042:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4990.829321] LustreError: 358042:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4990.858400] Lustre: server umount lustre-MDT0000 complete [ 4995.040805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4995.043229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4995.047317] Lustre: Skipped 1 previous similar message [ 4995.049226] LustreError: 355601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5000.299964] LustreError: 355579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5000.304667] LustreError: 355579:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5003.660146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5003.703243] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5003.798364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5003.800225] Lustre: Skipped 2 previous similar messages [ 5003.824682] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5005.030566] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5005.420418] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5008.869631] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5008.875977] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5008.893922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5010.054799] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5010.569356] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5012.420516] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5013.059908] Lustre: Failing over lustre-MDT0000 [ 5013.194164] LustreError: 359120:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5013.196020] LustreError: 359120:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5013.222742] Lustre: server umount lustre-MDT0000 complete [ 5013.986361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5013.988870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5013.990130] LustreError: 355601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5013.994852] Lustre: Skipped 4 previous similar messages [ 5013.998570] LustreError: 355601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5019.105365] LustreError: 355601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5019.110175] LustreError: 355601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5026.182446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5026.227065] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5026.381348] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5027.698562] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5031.020616] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5031.397547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5031.400731] Lustre: Skipped 2 previous similar messages [ 5031.408733] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5031.426195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5032.577519] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5033.112860] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5036.514947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.516264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5036.518120] Lustre: Skipped 2 previous similar messages [ 5036.521079] Lustre: Skipped 7 previous similar messages [ 5036.523984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5041.496906] Lustre: server umount lustre-MDT0000 complete [ 5041.633251] LustreError: 355601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5041.641229] LustreError: 355601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5042.701052] LustreError: 356300:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767039536 with bad export cookie 407129060576548161 [ 5042.705178] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5042.706057] LustreError: 356300:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5042.817167] Lustre: server umount lustre-MDT0001 complete [ 5053.234672] LustreError: 360550:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5053.236384] LustreError: 360550:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5053.280265] Lustre: server umount lustre-OST0000 complete [ 5058.354456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5058.512367] LustreError: 361377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5059.905823] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5062.756476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5064.205211] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5065.936352] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5067.104428] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5069.394874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5069.501615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5069.504806] Lustre: Skipped 3 previous similar messages [ 5071.340377] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5073.170778] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5084.903569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5090.274758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5090.274967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5090.275994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5090.276025] Lustre: Skipped 1 previous similar message [ 5090.277850] Lustre: Skipped 1 previous similar message [ 5096.209611] Lustre: server umount lustre-OST0000 complete [ 5103.573419] Lustre: server umount lustre-MDT0000 complete [ 5104.771589] LustreError: 361357:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767039598 with bad export cookie 407129060576548875 [ 5104.776698] LustreError: 361357:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5104.776957] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5104.898957] Lustre: server umount lustre-MDT0001 complete [ 5108.365171] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5109.296663] Key type lgssc unregistered [ 5109.398366] LNet: 364580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5109.401092] LNetError: 364580:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5109.407282] LNet: Removed LNI 192.168.204.105@tcp [ 5109.694920] Key type .llcrypt unregistered [ 5109.696152] Key type ._llcrypt unregistered [ 5119.704422] Key type ._llcrypt registered [ 5119.705341] Key type .llcrypt registered [ 5119.747834] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5120.070741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5120.101422] alg: No test for adler32 (adler32-zlib) [ 5120.957431] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 5121.035948] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 5122.616165] Key type lgssc registered [ 5122.961017] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5125.551684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5125.560190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5126.721420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5128.028532] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5131.034841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5131.186609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5132.604916] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5134.476750] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5135.660783] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5137.944886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5138.068761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5140.006117] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5141.745463] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5151.399537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5156.837078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5156.838718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5156.840293] Lustre: Skipped 2 previous similar messages [ 5156.840580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5156.842552] Lustre: Skipped 2 previous similar messages [ 5161.954282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5161.955967] Lustre: Skipped 2 previous similar messages [ 5162.372698] LustreError: 368322:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5162.402379] Lustre: server umount lustre-MDT0000 complete [ 5163.642102] LustreError: 366035:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767039657 with bad export cookie 7563382290165155477 [ 5163.645444] LustreError: 366035:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5163.646184] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5163.713242] LustreError: 368526:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5163.716605] LustreError: 368526:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5163.760804] Lustre: server umount lustre-MDT0001 complete [ 5174.195409] LustreError: 368726:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5174.197224] LustreError: 368726:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5174.246390] Lustre: server umount lustre-OST0000 complete [ 5178.616479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5178.760665] LustreError: 369456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5178.765985] LustreError: 369456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5178.786752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5180.031540] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5182.847601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5182.965549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5184.297831] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5186.113584] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5187.338462] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5189.638939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5191.536709] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5193.349384] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5204.134949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5218.786099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5218.789842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5218.795329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5219.813179] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5222.273406] LustreError: 371826:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5222.275729] LustreError: 371826:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5222.303961] Lustre: server umount lustre-MDT0000 complete [ 5223.548888] LustreError: 369436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767039717 with bad export cookie 7563382290165156576 [ 5223.553017] LustreError: 369436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5223.553918] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5223.681686] Lustre: server umount lustre-MDT0001 complete [ 5235.444109] LustreError: 372231:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5235.445959] LustreError: 372231:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5235.505923] Lustre: server umount lustre-OST0000 complete [ 5240.856487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5241.021154] LustreError: 373059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5241.044828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5241.047684] Lustre: Skipped 1 previous similar message [ 5242.369327] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5245.171448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5246.673274] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5248.496222] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5249.786366] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5252.053211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5254.083243] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5255.903345] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5265.511398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5271.011181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5271.012680] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5271.012962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5271.012967] Lustre: Skipped 3 previous similar messages [ 5271.014863] Lustre: Skipped 2 previous similar messages [ 5271.016908] LustreError: Skipped 1 previous similar message [ 5273.777135] LustreError: 375323:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5273.780159] LustreError: 375323:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5273.819557] Lustre: server umount lustre-OST0000 complete [ 5276.131065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5276.131502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5276.136871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5276.140903] Lustre: Skipped 2 previous similar messages [ 5281.232274] Lustre: server umount lustre-MDT0000 complete [ 5281.250164] LustreError: 373059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5281.254230] LustreError: 373059:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5282.450253] LustreError: 373040:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767039776 with bad export cookie 7563382290165157584 [ 5282.453615] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5282.454088] LustreError: 373040:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5282.565698] Lustre: server umount lustre-MDT0001 complete [ 5286.256743] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5287.239873] Key type lgssc unregistered [ 5287.354332] LNet: 376258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5287.357266] LNetError: 376258:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5287.367330] LNet: Removed LNI 192.168.204.105@tcp [ 5287.642759] Key type .llcrypt unregistered [ 5287.643775] Key type ._llcrypt unregistered [ 5297.138271] Key type ._llcrypt registered [ 5297.139307] Key type .llcrypt registered [ 5297.176432] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5297.620566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5297.627923] alg: No test for adler32 (adler32-zlib) [ 5298.477841] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 5298.559859] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 5300.136137] Key type lgssc registered [ 5300.483875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5303.042437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5303.048800] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5303.050854] LustreError: 377655:0:(osd_handler.c:8561:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5303.059175] LustreError: 377655:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5303.060907] LustreError: 377655:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5303.062661] LustreError: 377655:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5303.064691] LustreError: 377655:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5305.536239] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5305.538813] LustreError: 378053:0:(osd_handler.c:8561:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5305.546838] LustreError: 378053:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5305.548839] LustreError: 378053:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5305.550739] LustreError: 378053:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5305.553258] LustreError: 378053:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5311.870515] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5312.952118] Key type lgssc unregistered [ 5313.090489] LNet: 379037:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5313.093927] LNetError: 379037:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5313.104423] LNet: Removed LNI 192.168.204.105@tcp [ 5313.413438] Key type .llcrypt unregistered [ 5313.414342] Key type ._llcrypt unregistered [ 5319.153746] Key type ._llcrypt registered [ 5319.154753] Key type .llcrypt registered [ 5319.187151] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 5325.352426] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5325.809794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5325.821192] alg: No test for adler32 (adler32-zlib) [ 5326.668559] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 5326.752398] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 5328.328175] Key type lgssc registered [ 5328.722546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5331.792845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5333.971773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5335.963227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5338.079075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5340.412794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5340.439483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5340.445766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5341.534066] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5341.546792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5341.583223] Lustre: lustre-MDT0000: new disk, initializing [ 5341.609054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5341.616039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5342.890494] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5346.974482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5346.999956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5347.037827] Lustre: 382142:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5347.062154] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5347.065013] Lustre: Skipped 1 previous similar message [ 5347.107604] Lustre: lustre-MDT0001: new disk, initializing [ 5347.145630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5347.159806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5347.163464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5348.475928] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5351.561788] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5352.819897] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5355.134816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5355.160316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5355.260562] Lustre: lustre-OST0000: new disk, initializing [ 5355.262452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5355.291759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5357.259219] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5358.335124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5358.340373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5358.380423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5360.385556] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5361.880239] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5361.968300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5363.379631] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5363.451856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5368.802908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5368.803258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5368.807774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5368.809792] LustreError: Skipped 1 previous similar message [ 5368.813116] Lustre: Skipped 1 previous similar message [ 5370.157094] LustreError: 385079:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5370.196928] Lustre: server umount lustre-OST0000 complete [ 5372.896881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5372.898960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5372.901970] Lustre: Skipped 1 previous similar message [ 5372.903536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5373.921854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5373.921993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5377.539672] LustreError: 385281:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5377.542791] LustreError: 385281:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5377.573338] Lustre: server umount lustre-MDT0000 complete [ 5378.861375] LustreError: 382134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767039872 with bad export cookie 17343363663437279584 [ 5378.865429] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5378.866591] LustreError: 382134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5378.933674] LustreError: 385482:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5378.935775] LustreError: 385482:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5378.988528] Lustre: server umount lustre-MDT0001 complete [ 5381.896922] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5382.511773] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:24:35 (1767039875) [ 5384.755222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5384.926309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5386.213609] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5388.977635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5389.089200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5390.304978] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5392.052491] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5393.303590] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5395.549058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5397.652862] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5399.542909] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5401.882374] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5401.903710] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5401.960766] Lustre: lustre-OST0001: new disk, initializing [ 5401.963835] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5401.995194] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5401.998954] Lustre: Skipped 1 previous similar message [ 5403.984956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5403.988236] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5404.009490] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5404.047206] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5407.236745] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5409.252305] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5409.253166] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5409.254460] LustreError: Skipped 1 previous similar message [ 5409.258394] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5409.261255] Lustre: Skipped 2 previous similar messages [ 5413.493586] Lustre: lustre-OST0001: Not available for connect from 192.168.204.5@tcp (stopping) [ 5413.496648] Lustre: Skipped 2 previous similar messages [ 5415.151110] LustreError: 389399:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5415.152714] LustreError: 389399:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5415.172453] Lustre: server umount lustre-OST0001 complete [ 5415.395900] LustreError: 388284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5415.401628] LustreError: 388284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5420.519298] LustreError: 388284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5420.526590] LustreError: 388284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5425.637352] LustreError: 387799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5425.643977] LustreError: 387799:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5429.729129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5429.732627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5429.738224] Lustre: Skipped 1 previous similar message [ 5429.740777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5430.755897] LustreError: 387799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5430.762895] LustreError: 387799:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5433.393124] LustreError: 389623:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5433.396169] LustreError: 389623:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5433.437327] Lustre: server umount lustre-OST0000 complete [ 5435.360915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5435.364494] LustreError: Skipped 1 previous similar message [ 5446.113288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5446.115009] Lustre: Skipped 7 previous similar messages [ 5449.184143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5449.219992] LustreError: 389824:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5449.222635] LustreError: 389824:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5449.251398] Lustre: server umount lustre-MDT0000 complete [ 5450.495455] LustreError: 386181:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767039944 with bad export cookie 17343363663437280452 [ 5450.499576] LustreError: 386181:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5450.501255] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5456.691498] Lustre: server umount lustre-MDT0001 complete [ 5460.427379] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5461.398682] Key type lgssc unregistered [ 5461.523312] LNet: 390560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5461.525741] LNetError: 390560:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5461.537308] LNet: Removed LNI 192.168.204.105@tcp [ 5461.823509] Key type .llcrypt unregistered [ 5461.824673] Key type ._llcrypt unregistered [ 5472.841122] Key type ._llcrypt registered [ 5472.842106] Key type .llcrypt registered [ 5472.876481] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 5479.156640] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5479.710789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5479.738492] alg: No test for adler32 (adler32-zlib) [ 5480.613340] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 5480.706614] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 5482.288103] Key type lgssc registered [ 5482.668158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5485.463876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5487.654895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5489.740247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5491.821458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5494.174518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5494.202535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5494.206881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5495.300260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5495.311201] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5495.356654] Lustre: lustre-MDT0000: new disk, initializing [ 5495.384575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5495.391654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5496.696338] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5500.914384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5500.935732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5500.966786] Lustre: 394482:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5500.996341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5500.999374] Lustre: Skipped 1 previous similar message [ 5501.050915] Lustre: lustre-MDT0001: new disk, initializing [ 5501.089120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5501.101980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5501.105415] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5502.379868] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5505.505607] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5506.711154] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5508.951541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5508.979777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5509.078249] Lustre: lustre-OST0000: new disk, initializing [ 5509.081138] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5509.111321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5510.907443] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5511.864271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5511.867308] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5511.897748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5513.947697] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5515.429724] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5515.510835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5516.798447] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5516.875742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5522.406109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5522.407230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5522.409630] LustreError: Skipped 1 previous similar message [ 5522.414253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5522.417071] Lustre: Skipped 1 previous similar message [ 5523.633933] LustreError: 397421:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5523.680406] Lustre: server umount lustre-OST0000 complete [ 5527.008932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5527.010880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5527.013736] Lustre: Skipped 1 previous similar message [ 5527.015201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5532.641118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5532.642742] Lustre: Skipped 2 previous similar messages [ 5537.761162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5537.762783] Lustre: Skipped 1 previous similar message [ 5539.296110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5539.337545] LustreError: 397623:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5539.339461] LustreError: 397623:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5539.365384] Lustre: server umount lustre-MDT0000 complete [ 5540.611335] LustreError: 396228:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767040034 with bad export cookie 8139243090343827357 [ 5540.614823] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5540.615257] LustreError: 396228:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5540.686724] LustreError: 397827:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5540.689855] LustreError: 397827:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5540.754452] Lustre: server umount lustre-MDT0001 complete [ 5543.442592] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 15:27:16 (1767040036) [ 5546.403118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5549.095671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5549.244104] Lustre: lustre-MDT0000: new disk, initializing [ 5549.273627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5550.443585] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5553.089767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5554.482785] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5556.347754] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5557.459318] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5558.135586] LustreError: 399123:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5558.140085] LustreError: 399123:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5558.143788] LustreError: 399123:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 5558.243331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5558.244460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5558.245508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5558.250614] Lustre: Skipped 2 previous similar messages [ 5564.288309] LustreError: 400435:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5564.291367] LustreError: 400435:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5564.319612] Lustre: server umount lustre-MDT0000 complete [ 5565.553311] LustreError: 399074:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767040059 with bad export cookie 8139243090343828225 [ 5565.557229] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5565.682105] Lustre: server umount lustre-MDT0001 complete [ 5568.738887] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:27:41 (1767040061) [ 5570.970465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5571.137623] LustreError: 401356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5571.143305] LustreError: 401356:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5571.164994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5571.166820] Lustre: Skipped 1 previous similar message [ 5572.465426] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5575.260701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5576.715724] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5578.515458] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5579.719232] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5581.936507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5582.043535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5582.045675] Lustre: Skipped 1 previous similar message [ 5584.016142] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5585.855152] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5604.145792] Lustre: Permanently deactivating lustre-OST0000 [ 5604.150913] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5612.525641] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5613.773659] LustreError: 404167:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5613.776480] LustreError: 404167:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5613.778415] LustreError: 404167:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5614.395484] LustreError: 404215:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5614.397407] LustreError: 404215:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5621.101107] LustreError: 404312:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5621.103547] LustreError: 404312:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5621.145206] Lustre: server umount lustre-OST0000 complete [ 5622.024819] LustreError: 404464:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5622.028423] LustreError: 404464:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5626.848926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5626.850722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5626.853805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5626.856563] Lustre: Skipped 3 previous similar messages [ 5632.487605] LustreError: 404561:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5632.490815] LustreError: 404561:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5632.519138] Lustre: server umount lustre-MDT0000 complete [ 5633.778349] LustreError: 403725:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767040127 with bad export cookie 8139243090343828743 [ 5633.781104] LustreError: 403725:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5633.784546] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5643.290980] Lustre: server umount lustre-MDT0001 complete [ 5646.165794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5647.517961] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5648.466166] LustreError: 405604:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5650.559793] Lustre: 405942:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.105@tcp [ 5651.196529] LustreError: 406039:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5651.198265] LustreError: 406039:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5651.222678] Lustre: server umount MGS complete [ 5655.036402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5655.221795] LustreError: 406668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5655.228079] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5655.230674] Lustre: Skipped 1 previous similar message [ 5655.254343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5656.570781] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5659.409880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5659.513552] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5660.832846] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5662.545354] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5663.754135] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5664.421155] Lustre: Permanently reactivating lustre-OST0000 [ 5664.426209] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5664.428405] Lustre: Skipped 2 previous similar messages [ 5688.288145] LustreError: 408173:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5691.405132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5691.527826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5691.530174] Lustre: Skipped 1 previous similar message [ 5693.506121] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5695.219754] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5708.256793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5708.258940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5708.262029] Lustre: Skipped 1 previous similar message [ 5708.263612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5708.265288] Lustre: Skipped 4 previous similar messages [ 5713.197073] LustreError: 410004:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5713.199114] LustreError: 410004:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5713.233819] Lustre: server umount lustre-OST0000 complete [ 5719.523372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5719.526957] Lustre: Skipped 2 previous similar messages [ 5720.602964] Lustre: server umount lustre-MDT0000 complete [ 5721.894256] LustreError: 406648:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767040215 with bad export cookie 8139243090343829688 [ 5721.897387] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5721.899052] LustreError: 406648:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5722.022716] Lustre: server umount lustre-MDT0001 complete [ 5725.747724] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5726.762522] Key type lgssc unregistered [ 5726.883359] LNet: 410945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5726.885384] LNetError: 410945:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5726.894302] LNet: Removed LNI 192.168.204.105@tcp [ 5727.198581] Key type .llcrypt unregistered [ 5727.199563] Key type ._llcrypt unregistered [ 5733.053470] Key type ._llcrypt registered [ 5733.054441] Key type .llcrypt registered [ 5733.092869] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 5739.436423] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5739.848618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5739.934079] alg: No test for adler32 (adler32-zlib) [ 5740.798205] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 5740.898689] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 5742.488214] Key type lgssc registered [ 5742.897588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5745.947636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5748.072730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5750.038288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5752.186733] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5754.511852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5754.527094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5754.533776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5755.630897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5755.642987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5755.685494] Lustre: lustre-MDT0000: new disk, initializing [ 5755.711182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5755.717763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5757.032338] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5761.196093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5761.220574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5761.261514] Lustre: 414048:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5761.294734] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5761.296817] Lustre: Skipped 1 previous similar message [ 5761.336843] Lustre: lustre-MDT0001: new disk, initializing [ 5761.373130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5761.385674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5761.388985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5762.732435] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5765.770167] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5766.927605] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5769.167824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5769.193354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5769.294338] Lustre: lustre-OST0000: new disk, initializing [ 5769.297219] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5769.329546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5770.427437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5770.432223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5770.468215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5771.291614] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5774.468221] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5775.797873] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5775.875947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5777.244893] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5777.315920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5780.962390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5780.962517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5780.964464] LustreError: Skipped 1 previous similar message [ 5780.965431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5780.967861] Lustre: Skipped 1 previous similar message [ 5780.974621] Lustre: Skipped 1 previous similar message [ 5786.083592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5786.086591] Lustre: Skipped 1 previous similar message [ 5791.204536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5791.207600] Lustre: Skipped 1 previous similar message [ 5792.224077] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5792.240073] LustreError: 416984:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5792.277463] Lustre: server umount lustre-OST0000 complete [ 5796.322479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5796.323626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5796.325516] Lustre: Skipped 1 previous similar message [ 5796.329947] Lustre: Skipped 1 previous similar message [ 5799.619338] LustreError: 417187:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5799.621229] LustreError: 417187:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5799.646084] Lustre: server umount lustre-MDT0000 complete [ 5800.860323] LustreError: 414038:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767040294 with bad export cookie 14982941991938017559 [ 5800.863381] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5800.865927] LustreError: 414038:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5800.933337] LustreError: 417388:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5800.935016] LustreError: 417388:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5800.984552] Lustre: server umount lustre-MDT0001 complete [ 5803.746387] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5804.322971] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:31:37 (1767040297) [ 5806.540103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5806.722145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5808.062604] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5810.943717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5811.083600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5812.370389] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5814.175413] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5815.385723] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5817.745683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5819.770760] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5821.648806] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5822.383750] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5833.410636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5836.258480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5836.260979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5836.261361] LustreError: Skipped 1 previous similar message [ 5836.262037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5836.265223] Lustre: Skipped 1 previous similar message [ 5840.943098] LustreError: 420566:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5840.945023] LustreError: 420566:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5840.980752] Lustre: server umount lustre-OST0000 complete [ 5846.498291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5846.499259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5846.501259] Lustre: Skipped 1 previous similar message [ 5846.502663] Lustre: Skipped 1 previous similar message [ 5848.316800] LustreError: 420769:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5848.318390] LustreError: 420769:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5848.351038] Lustre: server umount lustre-MDT0000 complete [ 5849.600224] LustreError: 418087:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767040343 with bad export cookie 14982941991938018427 [ 5849.604292] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5849.605344] LustreError: 418087:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5849.732532] Lustre: server umount lustre-MDT0001 complete [ 5853.117791] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5854.111141] Key type lgssc unregistered [ 5854.242458] LNet: 421502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5854.245085] LNetError: 421502:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5854.254431] LNet: Removed LNI 192.168.204.105@tcp [ 5854.563610] Key type .llcrypt unregistered [ 5854.565305] Key type ._llcrypt unregistered [ 5862.085616] Key type ._llcrypt registered [ 5862.086644] Key type .llcrypt registered [ 5862.124788] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5862.217429] Key type .llcrypt unregistered [ 5862.218531] Key type ._llcrypt unregistered [ 5869.215221] Key type ._llcrypt registered [ 5869.216764] Key type .llcrypt registered [ 5869.259592] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5869.661204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5869.718119] alg: No test for adler32 (adler32-zlib) [ 5870.606330] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 5870.707166] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 5872.296138] Key type lgssc registered [ 5872.688433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5875.456788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5875.464906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5876.629650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5877.961334] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5880.760917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5880.881843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5882.739327] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5884.529792] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5885.922000] LustreError: 423646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5885.928826] LustreError: 423646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5886.725795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5886.846682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5887.913404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5888.092574] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5892.065748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5892.068572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5892.075498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5893.092062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5893.092256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5893.092576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5897.008093] LustreError: 425704:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5897.044124] Lustre: server umount lustre-OST0000 complete [ 5901.795043] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5901.798168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5901.803248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5901.805039] Lustre: Skipped 1 previous similar message [ 5904.450613] LustreError: 425905:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5904.452316] LustreError: 425905:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5904.476687] Lustre: server umount lustre-MDT0000 complete [ 5905.801736] LustreError: 425906:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767040399 with bad export cookie 7069502846990087637 [ 5905.807194] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5905.880506] LustreError: 426108:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5905.882941] LustreError: 426108:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5905.955060] Lustre: server umount lustre-MDT0001 complete [ 5909.653573] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5910.590911] Key type lgssc unregistered [ 5910.706434] LNet: 426639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5910.708299] LNetError: 426639:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5910.719315] LNet: Removed LNI 192.168.204.105@tcp [ 5911.008097] Key type .llcrypt unregistered [ 5911.009728] Key type ._llcrypt unregistered [ 5920.582698] Key type ._llcrypt registered [ 5920.583698] Key type .llcrypt registered [ 5920.626322] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5921.020201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5921.107730] alg: No test for adler32 (adler32-zlib) [ 5921.967105] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 5922.058179] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 5923.648185] Key type lgssc registered [ 5924.010065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5926.566466] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5926.573487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5942.048301] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5947.168150] LustreError: 427988:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.105@tcp: failed processing log, type 4: rc = -110 [ 5973.792152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5975.632809] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5977.278802] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5979.415400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5979.602246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5979.621807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5980.768836] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5983.627997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5983.766995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5984.946098] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5986.555678] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5987.633300] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5994.978495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5994.979537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5994.981126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5994.983708] Lustre: Skipped 1 previous similar message [ 6000.098100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6000.101404] Lustre: Skipped 2 previous similar messages [ 6000.239076] LustreError: 430271:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 6000.276423] Lustre: server umount lustre-OST0000 complete [ 6004.192811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6004.194799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6004.198055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6005.218206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6007.617927] LustreError: 430473:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6007.620774] LustreError: 430473:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6007.645102] Lustre: server umount lustre-MDT0000 complete [ 6008.818076] LustreError: 429624:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767040502 with bad export cookie 47411485999439220 [ 6008.820674] LustreError: 429624:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6008.821427] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6008.884547] LustreError: 430677:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6008.886158] LustreError: 430677:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6008.936866] Lustre: server umount lustre-MDT0001 complete [ 6012.305703] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 6013.332112] Key type lgssc unregistered [ 6013.463488] LNet: 431208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6013.465577] LNetError: 431208:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6013.474398] LNet: Removed LNI 192.168.204.105@tcp [ 6013.772612] Key type .llcrypt unregistered [ 6013.773725] Key type ._llcrypt unregistered [ 6023.006048] Key type ._llcrypt registered [ 6023.006907] Key type .llcrypt registered [ 6023.050789] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 6023.458790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6023.510139] alg: No test for adler32 (adler32-zlib) [ 6024.389147] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 6024.483921] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 6026.072132] Key type lgssc registered [ 6026.393035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6028.811089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6028.816536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6029.969576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6031.136664] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6033.878162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6034.014949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6035.339462] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6037.094634] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6038.292548] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6040.536777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6040.651114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6042.388022] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6044.078884] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6055.014946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6064.609720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6064.611529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6064.614856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6065.635302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6065.635783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6065.640603] Lustre: Skipped 1 previous similar message [ 6065.643803] Lustre: Skipped 3 previous similar messages [ 6066.685259] LustreError: 434847:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6066.715543] Lustre: server umount lustre-MDT0000 complete [ 6067.414673] LustreError: 434198:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.5@tcp arrived at 1767040561 with bad export cookie 15747582300750095033 [ 6067.420798] LustreError: 434198:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6068.015568] LustreError: 434198:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767040561 with bad export cookie 15747582300750094382 [ 6068.020115] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6069.728887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6069.732286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6069.737933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6070.753376] LustreError: 432580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6070.760258] LustreError: 432580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6073.760908] LustreError: 432580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.761429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6073.764912] LustreError: 432580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6073.766449] Lustre: Skipped 1 previous similar message [ 6074.094118] LustreError: 435050:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6074.095719] LustreError: 435050:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6074.127420] Lustre: server umount lustre-OST0000 complete [ 6076.157940] LustreError: 435404:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6076.159603] LustreError: 435404:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6076.222193] Lustre: server umount lustre-MDT0001 complete [ 6079.775124] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 6080.732740] Key type lgssc unregistered [ 6080.860562] LNet: 435935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6080.866153] LNetError: 435935:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6080.880478] LNet: Removed LNI 192.168.204.105@tcp [ 6081.181506] Key type .llcrypt unregistered [ 6081.182524] Key type ._llcrypt unregistered [ 6090.371884] Key type ._llcrypt registered [ 6090.373554] Key type .llcrypt registered [ 6090.417395] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 6090.873821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6090.882062] alg: No test for adler32 (adler32-zlib) [ 6091.729920] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 6091.807117] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 6093.384189] Key type lgssc registered [ 6093.717817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6096.211284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6096.218427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6097.410303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6098.725765] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6101.459696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6101.591169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6102.839535] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6104.521059] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6105.610249] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6107.792223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6107.907836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6109.719709] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6111.453652] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6122.280384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6132.193058] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6132.194933] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6132.198338] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6132.707786] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6132.714162] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6132.716836] Lustre: Skipped 1 previous similar message [ 6134.208239] LustreError: 439574:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6134.231407] Lustre: server umount lustre-MDT0001 complete [ 6137.825968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6137.827082] LustreError: 438031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6137.827345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6137.828687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6137.831428] LustreError: 438031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6141.935125] LustreError: 439776:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6141.938176] LustreError: 439776:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6141.974952] Lustre: server umount lustre-OST0000 complete [ 6142.946892] LustreError: 438031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6143.268564] LustreError: 439978:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6143.271613] LustreError: 439978:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6143.326255] Lustre: server umount lustre-MDT0000 complete [ 6147.930143] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 6148.930737] Key type lgssc unregistered [ 6149.042432] LNet: 440664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6149.044418] LNetError: 440664:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6149.057342] LNet: Removed LNI 192.168.204.105@tcp [ 6149.364808] Key type .llcrypt unregistered [ 6149.365932] Key type ._llcrypt unregistered [ 6157.298500] Key type ._llcrypt registered [ 6157.299525] Key type .llcrypt registered [ 6157.340727] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 6163.350685] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 6163.776055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6163.787156] alg: No test for adler32 (adler32-zlib) [ 6164.629974] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 6164.707165] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 6166.280132] Key type lgssc registered [ 6166.596082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6169.255113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6171.138692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6172.928974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6174.841933] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6177.969683] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 6180.648515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6182.706874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6182.722849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6182.728403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6183.809803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6183.820858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6183.857549] Lustre: lustre-MDT0000: new disk, initializing [ 6183.882267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6183.888650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6185.045730] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6188.344557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6190.427197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6190.453445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6190.543271] Lustre: lustre-OST0000: new disk, initializing [ 6190.545170] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6190.546777] Lustre: Skipped 1 previous similar message [ 6190.567583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6192.344479] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6195.239282] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6196.219867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6196.222123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6196.229128] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6203.386025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6205.447643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6205.471164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6205.501719] Lustre: 444609:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6205.525743] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6205.560594] Lustre: lustre-MDT0001: new disk, initializing [ 6205.599614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6205.612246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6205.615312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6206.743900] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6210.105161] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6210.168598] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6211.335626] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6213.442291] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [ 6217.843593] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [ 6218.212578] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6218.212586] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6218.223126] Lustre: Skipped 1 previous similar message [ 6221.053292] LustreError: 448042:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6221.077579] Lustre: server umount lustre-MDT0001 complete [ 6228.450052] LustreError: 444614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6228.454220] LustreError: 444614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6233.569107] LustreError: 444618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.569420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6233.577691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6233.579684] Lustre: Skipped 2 previous similar messages [ 6238.689241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6238.689821] LustreError: 444618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6239.279150] LustreError: 448316:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6239.280842] LustreError: 448316:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6239.315529] Lustre: server umount lustre-OST0000 complete [ 6246.719744] LustreError: 448517:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6246.721466] LustreError: 448517:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6246.773596] Lustre: server umount lustre-MDT0000 complete [ 6250.767606] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 6251.634633] Key type lgssc unregistered [ 6251.752299] LNet: 449201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6251.754332] LNetError: 449201:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6251.761377] LNet: Removed LNI 192.168.204.105@tcp [ 6252.029490] Key type .llcrypt unregistered [ 6252.030395] Key type ._llcrypt unregistered [ 6263.699509] Key type ._llcrypt registered [ 6263.700504] Key type .llcrypt registered [ 6263.733680] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 6263.818275] Key type .llcrypt unregistered [ 6263.819159] Key type ._llcrypt unregistered [ 6269.934685] Key type ._llcrypt registered [ 6269.935572] Key type .llcrypt registered [ 6269.967607] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 6270.289870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6270.323871] alg: No test for adler32 (adler32-zlib) [ 6271.159219] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 6271.230855] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 6272.808099] Key type lgssc registered [ 6273.111260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6275.313193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6277.157213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6278.908122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6280.895196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6280.914813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6280.918377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6281.992680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6282.001898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6282.035795] Lustre: lustre-MDT0000: new disk, initializing [ 6282.057235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6282.062662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6283.257338] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6287.005533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6287.028245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6287.057400] Lustre: 452519:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6287.077674] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6287.079071] Lustre: Skipped 1 previous similar message [ 6287.112657] Lustre: lustre-MDT0001: new disk, initializing [ 6287.141492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6287.151460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6287.154186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6288.254559] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6291.053275] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6292.096124] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6294.059883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6294.079304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6294.163434] Lustre: lustre-OST0000: new disk, initializing [ 6294.164973] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6294.189929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6295.543095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6295.547908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6295.578376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6295.930137] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6298.761510] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6302.688908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6302.691294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6302.694875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6305.762340] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6305.763706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6305.765248] Lustre: Skipped 1 previous similar message [ 6305.768371] Lustre: Skipped 1 previous similar message [ 6307.151480] LustreError: 454959:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6307.185817] Lustre: server umount lustre-MDT0000 complete [ 6309.009837] LustreError: 453299:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767040802 with bad export cookie 11203281822350928148 [ 6309.016555] LustreError: 453299:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6309.017274] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6309.109389] LustreError: 455160:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6309.112355] LustreError: 455160:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6309.171683] Lustre: server umount lustre-MDT0001 complete [ 6321.078514] LustreError: 455361:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6321.082344] LustreError: 455361:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6321.148178] Lustre: server umount lustre-OST0000 complete [ 6325.097560] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 15:40:17 (1767040817) [ 6325.520672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6328.244674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6328.416790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6329.640238] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6333.376734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6333.529842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6334.797686] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6337.101403] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6338.312902] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6340.678435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6340.722700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6340.836386] Lustre: Found index 0 for lustre-OST0000, updating log [ 6343.431490] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6345.257608] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6354.279270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6359.521577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6359.523076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6359.523450] LustreError: Skipped 1 previous similar message [ 6359.524128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6359.526439] Lustre: Skipped 1 previous similar message [ 6363.313728] LustreError: 458466:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6363.316674] LustreError: 458466:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6363.368252] Lustre: server umount lustre-OST0000 complete [ 6364.231150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6365.665834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6365.666471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6365.671964] Lustre: Skipped 1 previous similar message [ 6365.673255] Lustre: Skipped 3 previous similar messages [ 6370.968103] LustreError: 458721:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6370.971423] LustreError: 458721:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6371.006066] Lustre: server umount lustre-MDT0000 complete [ 6371.809226] LustreError: 456206:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6371.813270] LustreError: 456206:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6372.332649] LustreError: 458723:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767040866 with bad export cookie 11203281822350929527 [ 6372.338375] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6372.338896] LustreError: 458723:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6372.470771] Lustre: server umount lustre-MDT0001 complete [ 6375.122665] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 15:41:08 (1767040868) [ 6375.426461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6375.725828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6376.040345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6376.355979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6376.699215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6377.005321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6377.334206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6377.664808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6377.985044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6378.407571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6378.860067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6379.151298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6379.464520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6379.780317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6380.095241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6380.415569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6380.743651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6381.124371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6381.604152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6382.080309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6382.564642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6383.039046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6383.500691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6383.984187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6384.300210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6384.617291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6384.923194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6385.215917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6385.510038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6385.811183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6386.097066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6386.484492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6386.938766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6387.414807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6387.915380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6388.230201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6388.540668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6388.989700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6389.300935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6389.608958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6389.901893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6390.189941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6390.480086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6390.814376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6391.246530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6391.672978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6392.008401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6392.458446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6392.943216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6393.446760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6393.925205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6396.976845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6397.190464] LustreError: 462305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6397.218289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6397.222659] Lustre: Skipped 1 previous similar message [ 6398.957839] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6402.434030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6403.858822] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6406.364291] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6407.456466] LNet: 1 local NIs in recovery (showing 1): 192.168.204.105@tcp [ 6408.184227] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6411.441914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6411.484402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6411.591163] Lustre: Found index 0 for lustre-OST0000, updating log [ 6413.896816] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6415.779554] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6418.920086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6418.921389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6418.931132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6418.934644] Lustre: Skipped 3 previous similar messages [ 6422.772985] LustreError: 464614:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6422.777231] LustreError: 464614:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6422.842248] Lustre: server umount lustre-OST0000 complete [ 6427.746975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6427.748477] Lustre: Skipped 4 previous similar messages [ 6430.810078] Lustre: server umount lustre-MDT0000 complete [ 6430.818399] LustreError: 463514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6430.822935] LustreError: 463514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6432.297782] LustreError: 462285:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767040926 with bad export cookie 11203281822350930598 [ 6432.302088] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6432.304086] LustreError: 462285:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6432.445639] Lustre: server umount lustre-MDT0001 complete [ 6433.790990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6440.358304] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 6444.600929] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 6448.742240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6450.944455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6453.770897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6456.952519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6460.123684] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 15:42:32 (1767040952) [ 6467.872108] LNet: 1 local NIs in recovery (showing 1): 192.168.204.105@tcp [ 6478.910182] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 6479.996943] Key type lgssc unregistered [ 6480.107468] LNet: 468608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6480.109744] LNetError: 468608:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6608.105187] LNet: Removed LNI 192.168.204.105@tcp [ 6608.508550] Key type .llcrypt unregistered [ 6608.510228] Key type ._llcrypt unregistered [ 6615.987036] Key type ._llcrypt registered [ 6615.987886] Key type .llcrypt registered [ 6616.019708] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 6616.296778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6616.302622] alg: No test for adler32 (adler32-zlib) [ 6617.175380] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 6617.302298] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 6618.912215] Key type lgssc registered [ 6619.504761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6620.850680] LNet: Added LNI 192.168.204.105@tcp100 [8/256/0/180] [ 6621.116872] LNet: Added LNI 192.168.204.105@tcp101 [8/256/0/180] [ 6621.383141] LNet: Added LNI 192.168.204.105@tcp102 [8/256/0/180] [ 6621.684477] LNet: Added LNI 192.168.204.105@tcp103 [8/256/0/180] [ 6622.010381] LNet: Added LNI 192.168.204.105@tcp104 [8/256/0/180] [ 6622.397405] LNet: Added LNI 192.168.204.105@tcp105 [8/256/0/180] [ 6622.836479] LNet: Added LNI 192.168.204.105@tcp106 [8/256/0/180] [ 6623.262480] LNet: Added LNI 192.168.204.105@tcp107 [8/256/0/180] [ 6623.677273] LNet: Added LNI 192.168.204.105@tcp108 [8/256/0/180] [ 6624.108898] LNet: Added LNI 192.168.204.105@tcp109 [8/256/0/180] [ 6624.536214] LNet: Added LNI 192.168.204.105@tcp110 [8/256/0/180] [ 6624.804100] LNet: Added LNI 192.168.204.105@tcp111 [8/256/0/180] [ 6625.070035] LNet: Added LNI 192.168.204.105@tcp112 [8/256/0/180] [ 6625.339484] LNet: Added LNI 192.168.204.105@tcp113 [8/256/0/180] [ 6625.620889] LNet: Added LNI 192.168.204.105@tcp114 [8/256/0/180] [ 6625.890567] LNet: Added LNI 192.168.204.105@tcp115 [8/256/0/180] [ 6626.155349] LNet: Added LNI 192.168.204.105@tcp116 [8/256/0/180] [ 6626.427182] LNet: Added LNI 192.168.204.105@tcp117 [8/256/0/180] [ 6626.751944] LNet: Added LNI 192.168.204.105@tcp118 [8/256/0/180] [ 6627.194782] LNet: Added LNI 192.168.204.105@tcp119 [8/256/0/180] [ 6627.627839] LNet: Added LNI 192.168.204.105@tcp120 [8/256/0/180] [ 6628.058523] LNet: Added LNI 192.168.204.105@tcp121 [8/256/0/180] [ 6628.506396] LNet: Added LNI 192.168.204.105@tcp122 [8/256/0/180] [ 6628.935632] LNet: Added LNI 192.168.204.105@tcp123 [8/256/0/180] [ 6629.377810] LNet: Added LNI 192.168.204.105@tcp124 [8/256/0/180] [ 6629.767414] LNet: Added LNI 192.168.204.105@tcp125 [8/256/0/180] [ 6630.050148] LNet: Added LNI 192.168.204.105@tcp126 [8/256/0/180] [ 6630.321678] LNet: Added LNI 192.168.204.105@tcp127 [8/256/0/180] [ 6630.595462] LNet: Added LNI 192.168.204.105@tcp128 [8/256/0/180] [ 6630.865510] LNet: Added LNI 192.168.204.105@tcp129 [8/256/0/180] [ 6631.142461] LNet: Added LNI 192.168.204.105@tcp130 [8/256/0/180] [ 6631.501506] LNet: Added LNI 192.168.204.105@tcp131 [8/256/0/180] [ 6631.918666] LNet: Added LNI 192.168.204.105@tcp132 [8/256/0/180] [ 6632.366331] LNet: Added LNI 192.168.204.105@tcp133 [8/256/0/180] [ 6632.786386] LNet: Added LNI 192.168.204.105@tcp134 [8/256/0/180] [ 6633.208924] LNet: Added LNI 192.168.204.105@tcp135 [8/256/0/180] [ 6635.839451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6635.863154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6635.867228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6636.911546] Lustre: 471596:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.105@tcp: can't add NID 192.168.204.105@tcp130: rc = -75 [ 6637.008625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6637.024447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6637.073968] Lustre: lustre-MDT0000: new disk, initializing [ 6637.119242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6637.130092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6639.031760] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6643.778754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6643.815246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6643.871044] Lustre: 471614:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6643.914482] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6643.918268] Lustre: Skipped 1 previous similar message [ 6643.974905] Lustre: lustre-MDT0001: new disk, initializing [ 6643.989210] Lustre: 472400:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.105@tcp: can't add NID 192.168.204.105@tcp130: rc = -75 [ 6643.992342] Lustre: 472400:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6644.038538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6644.055183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6644.059477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6645.608815] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6649.065910] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6649.867906] Lustre: 472958:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.105@tcp: can't add NID 192.168.204.105@tcp130: rc = -75 [ 6649.873935] Lustre: 472958:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6650.617235] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6652.924942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6652.955264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6653.094764] Lustre: lustre-OST0000: new disk, initializing [ 6653.096621] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6653.128749] Lustre: 473373:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.105@tcp: can't add NID 192.168.204.105@tcp130: rc = -75 [ 6653.132388] Lustre: 473373:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 6653.148900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6655.053443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6655.057067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6655.115306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6655.524992] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6658.792352] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6665.698082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6665.699671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6665.699929] LustreError: Skipped 1 previous similar message [ 6665.704109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6665.705976] Lustre: Skipped 1 previous similar message [ 6667.631125] LustreError: 474207:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6667.678687] Lustre: server umount lustre-OST0000 complete [ 6669.792854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6669.795188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.800474] Lustre: Skipped 1 previous similar message [ 6669.803176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6670.817174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6670.818404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6675.602336] LustreError: 474410:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6675.608403] LustreError: 474410:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6675.646021] Lustre: server umount lustre-MDT0000 complete [ 6676.909312] LustreError: 471606:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767041170 with bad export cookie 5989288571821381849 [ 6676.912338] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6676.914756] LustreError: 471606:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6676.987528] LustreError: 474612:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6676.989072] LustreError: 474612:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6677.033549] Lustre: server umount lustre-MDT0001 complete [ 6681.275576] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 6682.119580] Key type lgssc unregistered [ 6682.261499] LNet: 475152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6682.263920] LNetError: 475152:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6682.265679] LNet: Removed LNI 192.168.204.105@tcp [ 6683.296137] LNet: Removed LNI 192.168.204.105@tcp100 [ 6683.297598] LNetError: 475152:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6683.299164] LNetError: 475152:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6683.301645] LNet: Removed LNI 192.168.204.105@tcp101 [ 6684.320175] LNet: Removed LNI 192.168.204.105@tcp102 [ 6684.322692] LNetError: 475152:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6684.326909] LNetError: 475152:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6684.331436] LNet: Removed LNI 192.168.204.105@tcp103 [ 6685.344221] LNet: Removed LNI 192.168.204.105@tcp104 [ 6686.368099] LNet: Removed LNI 192.168.204.105@tcp105 [ 6686.369553] LNetError: 475152:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6686.371881] LNetError: 475152:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 6686.374278] LNet: Removed LNI 192.168.204.105@tcp106 [ 6687.392161] LNet: Removed LNI 192.168.204.105@tcp107 [ 6688.416180] LNet: Removed LNI 192.168.204.105@tcp108 [ 6689.440161] LNet: Removed LNI 192.168.204.105@tcp109 [ 6690.464150] LNet: Removed LNI 192.168.204.105@tcp110 [ 6690.467221] LNetError: 475152:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6690.472340] LNetError: 475152:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 6690.477860] LNet: Removed LNI 192.168.204.105@tcp111 [ 6691.488190] LNet: Removed LNI 192.168.204.105@tcp112 [ 6691.490950] LNet: Removed LNI 192.168.204.105@tcp113 [ 6691.493491] LNet: Removed LNI 192.168.204.105@tcp114 [ 6691.496729] LNet: Removed LNI 192.168.204.105@tcp115 [ 6691.500083] LNet: Removed LNI 192.168.204.105@tcp116 [ 6691.503269] LNet: Removed LNI 192.168.204.105@tcp117 [ 6691.506896] LNet: Removed LNI 192.168.204.105@tcp118 [ 6691.510085] LNet: Removed LNI 192.168.204.105@tcp119 [ 6691.512585] LNet: Removed LNI 192.168.204.105@tcp120 [ 6691.516375] LNet: Removed LNI 192.168.204.105@tcp121 [ 6691.520389] LNet: Removed LNI 192.168.204.105@tcp122 [ 6691.524692] LNet: Removed LNI 192.168.204.105@tcp123 [ 6691.527534] LNet: Removed LNI 192.168.204.105@tcp124 [ 6691.530163] LNet: Removed LNI 192.168.204.105@tcp125 [ 6691.533460] LNet: Removed LNI 192.168.204.105@tcp126 [ 6691.536662] LNet: Removed LNI 192.168.204.105@tcp127 [ 6691.539775] LNet: Removed LNI 192.168.204.105@tcp128 [ 6691.542260] LNet: Removed LNI 192.168.204.105@tcp129 [ 6691.544525] LNet: Removed LNI 192.168.204.105@tcp130 [ 6691.546780] LNet: Removed LNI 192.168.204.105@tcp131 [ 6691.549598] LNet: Removed LNI 192.168.204.105@tcp132 [ 6691.551586] LNet: Removed LNI 192.168.204.105@tcp133 [ 6691.553201] LNet: Removed LNI 192.168.204.105@tcp134 [ 6691.560544] LNet: Removed LNI 192.168.204.105@tcp135 [ 6691.958242] Key type .llcrypt unregistered [ 6691.959435] Key type ._llcrypt unregistered [ 6701.314162] Key type ._llcrypt registered [ 6701.315727] Key type .llcrypt registered [ 6701.372595] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 6709.404189] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 6709.743105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6709.784750] alg: No test for adler32 (adler32-zlib) [ 6710.646663] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 6710.728992] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 6712.304158] Key type lgssc registered [ 6712.875776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6717.259123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6720.572379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6723.638818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6726.904903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6730.063345] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 15:47:03 (1767041223) [ 6737.278911] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 6738.873340] Key type lgssc unregistered [ 6739.045538] LNet: 479167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6739.049896] LNetError: 479167:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6739.060750] LNet: Removed LNI 192.168.204.105@tcp [ 6739.515640] Key type .llcrypt unregistered [ 6739.517236] Key type ._llcrypt unregistered [ 6748.355871] Key type ._llcrypt registered [ 6748.357106] Key type .llcrypt registered [ 6748.421786] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 6748.811858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6748.886486] alg: No test for adler32 (adler32-zlib) [ 6749.796820] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 6749.895537] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 6751.472183] Key type lgssc registered [ 6752.089920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6756.788829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6756.824493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6756.833423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.969810] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6757.990133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6758.046465] Lustre: lustre-MDT0000: new disk, initializing [ 6758.091254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6758.103392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6760.068943] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6765.577745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6765.601375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6765.636264] Lustre: 480338:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6765.667484] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6765.669114] Lustre: Skipped 1 previous similar message [ 6765.703908] Lustre: lustre-MDT0001: new disk, initializing [ 6765.756014] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6765.776722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6765.782690] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6767.445898] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6770.467515] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6771.792745] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6774.109939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6774.138535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6774.234714] Lustre: lustre-OST0000: new disk, initializing [ 6774.236816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6774.268683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6776.181895] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6779.177273] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6780.443786] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing lnet_if_list [ 6781.950550] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 6781.959360] LNet: Added LNI 192.168.204.105@tcp10 [8/256/0/180] [ 6782.975206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6782.978331] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6783.028428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6783.513411] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 6783.525317] LNet: Added LNI 192.168.204.105@tcp11 [8/256/0/180] [ 6784.972119] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 6784.982355] LNet: Added LNI 192.168.204.105@tcp12 [8/256/0/180] [ 6786.168298] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 6786.177712] LNet: Added LNI 192.168.204.105@tcp13 [8/256/0/180] [ 6787.371257] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 6787.380787] LNet: Added LNI 192.168.204.105@tcp14 [8/256/0/180] [ 6788.895129] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 6788.903595] LNet: Added LNI 192.168.204.105@tcp15 [8/256/0/180] [ 6790.162485] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 6790.170835] LNet: Added LNI 192.168.204.105@tcp16 [8/256/0/180] [ 6791.449854] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 6791.460965] LNet: Added LNI 192.168.204.105@tcp17 [8/256/0/180] [ 6793.102496] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 6793.111128] LNet: Added LNI 192.168.204.105@tcp18 [8/256/0/180] [ 6794.524351] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 6794.533360] LNet: Added LNI 192.168.204.105@tcp19 [8/256/0/180] [ 6796.173353] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 6796.189962] LNet: Added LNI 192.168.204.105@tcp20 [8/256/0/180] [ 6797.743292] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 6797.752015] LNet: Added LNI 192.168.204.105@tcp21 [8/256/0/180] [ 6799.220612] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 6799.229693] LNet: Added LNI 192.168.204.105@tcp22 [8/256/0/180] [ 6801.055343] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 6801.066241] LNet: Added LNI 192.168.204.105@tcp23 [8/256/0/180] [ 6802.618885] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 6802.628274] LNet: Added LNI 192.168.204.105@tcp24 [8/256/0/180] [ 6804.357330] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 6804.372185] LNet: Added LNI 192.168.204.105@tcp25 [8/256/0/180] [ 6806.165420] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 6806.174695] LNet: Added LNI 192.168.204.105@tcp26 [8/256/0/180] [ 6808.061554] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 6808.080093] LNet: Added LNI 192.168.204.105@tcp27 [8/256/0/180] [ 6808.095242] LNetError: 479600:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.105 [ 6808.100416] LNetError: Protocol error connecting to 192.168.204.105@tcp26 at host 192.168.204.105:988: is it running a compatible version of Lustre? [ 6810.720729] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 6810.739075] LNet: Added LNI 192.168.204.105@tcp28 [8/256/0/180] [ 6810.746903] LNetError: 479597:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.105 [ 6810.753215] LNetError: Protocol error connecting to 192.168.204.105@tcp26 at host 192.168.204.105:988: is it running a compatible version of Lustre? [ 6810.759823] LNetError: Skipped 1 previous similar message [ 6813.095058] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 6813.104443] LNet: Added LNI 192.168.204.105@tcp29 [8/256/0/180] [ 6813.108776] LNetError: 479600:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.105 [ 6813.111577] LNetError: 479600:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [ 6813.113591] LNetError: Protocol error connecting to 192.168.204.105@tcp27 at host 192.168.204.105:988: is it running a compatible version of Lustre? [ 6813.116260] LNetError: Skipped 1 previous similar message [ 6815.346010] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 6815.364853] LNet: Added LNI 192.168.204.105@tcp30 [8/256/0/180] [ 6815.372147] LNetError: 479597:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.105 [ 6815.378388] LNetError: 479597:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [ 6815.384900] LNetError: Protocol error connecting to 192.168.204.105@tcp28 at host 192.168.204.105:988: is it running a compatible version of Lustre? [ 6815.389982] LNetError: Skipped 1 previous similar message [ 6818.149813] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 6818.162618] LNet: Added LNI 192.168.204.105@tcp31 [8/256/0/180] [ 6818.529101] LNet: 1 local NIs in recovery (showing 1): 192.168.204.105@tcp30 [ 6819.553331] LNetError: 479599:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.105 [ 6819.556333] LNetError: 479599:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 6819.558647] LNetError: Protocol error connecting to 192.168.204.105@tcp30 at host 192.168.204.105:988: is it running a compatible version of Lustre? [ 6819.562836] LNetError: Skipped 2 previous similar messages [ 6820.339589] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 6820.356862] LNet: Added LNI 192.168.204.105@tcp32 [8/256/0/180] [ 6823.222122] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 6823.232354] LNet: Added LNI 192.168.204.105@tcp33 [8/256/0/180] [ 6825.616335] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 6825.638395] LNet: Added LNI 192.168.204.105@tcp34 [8/256/0/180] [ 6827.748162] LNetError: 479600:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.105 [ 6827.751628] LNetError: 479600:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 9 previous similar messages [ 6827.754463] LNetError: Protocol error connecting to 192.168.204.105@tcp33 at host 192.168.204.105:988: is it running a compatible version of Lustre? [ 6827.758512] LNetError: Skipped 9 previous similar messages [ 6828.458164] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 6828.471696] LNet: Added LNI 192.168.204.105@tcp35 [8/256/0/180] [ 6831.016406] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 6831.035353] LNet: Added LNI 192.168.204.105@tcp36 [8/256/0/180] [ 6833.677788] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 6833.687427] LNet: Added LNI 192.168.204.105@tcp37 [8/256/0/180] [ 6836.144044] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 6836.156322] LNet: Added LNI 192.168.204.105@tcp38 [8/256/0/180] [ 6837.232748] LNetError: 479607:0:(peer.c:1886:lnet_peer_add_nid()) Peer 192.168.204.105@tcp NID 192.168.204.105@tcp is already locked with peer 192.168.204.105@tcp [ 6838.670221] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 6838.679586] LNet: Added LNI 192.168.204.105@tcp39 [8/256/0/180] [ 6838.680906] LNetError: 479607:0:(lib-move.c:2321:lnet_handle_find_routed_path()) peer 192.168.204.105@tcp has no available nets [ 6841.175271] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 6841.192384] LNet: Added LNI 192.168.204.105@tcp40 [8/256/0/180] [ 6841.195640] LNetError: 479607:0:(lib-move.c:2321:lnet_handle_find_routed_path()) peer 192.168.204.105@tcp has no available nets [ 6841.200156] LNetError: 479607:0:(lib-move.c:2321:lnet_handle_find_routed_path()) Skipped 1 previous similar message [ 6843.922832] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 6843.933448] LNet: Added LNI 192.168.204.105@tcp41 [8/256/0/180] [ 6844.130060] LNetError: 479599:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.105 [ 6844.132434] LNetError: 479599:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 28 previous similar messages [ 6844.134336] LNetError: Protocol error connecting to 192.168.204.105@tcp39 at host 192.168.204.105:988: is it running a compatible version of Lustre? [ 6844.136605] LNetError: Skipped 29 previous similar messages [ 6846.478337] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 6846.488499] LNet: Added LNI 192.168.204.105@tcp42 [8/256/0/180] [ 6848.897565] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 6848.905888] LNet: Added LNI 192.168.204.105@tcp43 [8/256/0/180] [ 6851.493336] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 6851.508762] LNet: Added LNI 192.168.204.105@tcp44 [8/256/0/180] [ 6854.160941] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 6854.172313] LNet: Added LNI 192.168.204.105@tcp45 [8/256/0/180] [ 6856.333193] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 6856.341947] LNet: Added LNI 192.168.204.105@tcp46 [8/256/0/180] [ 6858.462312] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 6858.472038] LNet: Added LNI 192.168.204.105@tcp47 [8/256/0/180] [ 6860.812483] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 6860.829880] LNet: Added LNI 192.168.204.105@tcp48 [8/256/0/180] [ 6863.632070] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 6863.640996] LNet: Added LNI 192.168.204.105@tcp49 [8/256/0/180] [ 6866.478205] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 6866.493100] LNet: Added LNI 192.168.204.105@tcp50 [8/256/0/180] [ 6875.106338] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6875.107263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6875.107908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6875.108960] LustreError: Skipped 1 previous similar message [ 6875.114292] Lustre: Skipped 1 previous similar message [ 6876.897722] LNetError: 479599:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.105 [ 6876.903489] LNetError: 479599:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 46 previous similar messages [ 6876.908108] LNetError: Protocol error connecting to 192.168.204.105@tcp29 at host 192.168.204.105:988: is it running a compatible version of Lustre? [ 6876.914021] LNetError: Skipped 45 previous similar messages [ 6878.944188] LNet: 9 local NIs in recovery (showing 9): 192.168.204.105@tcp45, 192.168.204.105@tcp44, 192.168.204.105@tcp48, 192.168.204.105@tcp43, 192.168.204.105@tcp36, 192.168.204.105@tcp33, 192.168.204.105@tcp32, 192.168.204.105@tcp31, 192.168.204.105@tcp30 [ 6879.996353] LustreError: 504522:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6880.056307] Lustre: server umount lustre-OST0000 complete [ 6883.808730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6883.811244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6883.815111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6883.817280] Lustre: Skipped 1 previous similar message [ 6885.346343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6885.347282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6890.466126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6890.469682] Lustre: Skipped 2 previous similar messages [ 6895.586384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6895.591609] Lustre: Skipped 1 previous similar message [ 6896.096117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6896.149823] LustreError: 504723:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6896.154610] LustreError: 504723:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6896.192249] Lustre: server umount lustre-MDT0000 complete [ 6897.923964] LustreError: 480328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767041391 with bad export cookie 2769896471473037285 [ 6897.926474] LustreError: 480328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6897.927608] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6897.989585] LustreError: 504928:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6897.992198] LustreError: 504928:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6898.050211] Lustre: server umount lustre-MDT0001 complete [ 6901.788681] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 6903.224924] Key type lgssc unregistered [ 6903.401756] LNet: 505467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6903.405942] LNetError: 505467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6903.409388] LNet: Removed LNI 192.168.204.105@tcp [ 6904.416212] LNet: Removed LNI 192.168.204.105@tcp10 [ 6904.418928] LNetError: 505467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6904.423783] LNetError: 505467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6904.428334] LNet: Removed LNI 192.168.204.105@tcp11 [ 6905.440165] LNet: Removed LNI 192.168.204.105@tcp12 [ 6905.442523] LNetError: 505467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6905.446424] LNetError: 505467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6905.449879] LNet: Removed LNI 192.168.204.105@tcp13 [ 6906.464126] LNet: Removed LNI 192.168.204.105@tcp14 [ 6907.488109] LNet: Removed LNI 192.168.204.105@tcp15 [ 6907.489672] LNetError: 505467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6907.493379] LNetError: 505467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 6907.496841] LNet: Removed LNI 192.168.204.105@tcp16 [ 6908.512190] LNet: Removed LNI 192.168.204.105@tcp17 [ 6909.536258] LNet: Removed LNI 192.168.204.105@tcp18 [ 6910.560217] LNet: Removed LNI 192.168.204.105@tcp19 [ 6911.584220] LNet: Removed LNI 192.168.204.105@tcp20 [ 6911.586704] LNetError: 505467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6911.591572] LNetError: 505467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 6911.596836] LNet: Removed LNI 192.168.204.105@tcp21 [ 6912.608242] LNet: Removed LNI 192.168.204.105@tcp22 [ 6913.632205] LNet: Removed LNI 192.168.204.105@tcp23 [ 6914.656205] LNet: Removed LNI 192.168.204.105@tcp24 [ 6915.680145] LNet: Removed LNI 192.168.204.105@tcp25 [ 6916.704196] LNet: Removed LNI 192.168.204.105@tcp26 [ 6917.728205] LNet: Removed LNI 192.168.204.105@tcp27 [ 6918.752186] LNet: Removed LNI 192.168.204.105@tcp28 [ 6919.776223] LNet: Removed LNI 192.168.204.105@tcp29 [ 6919.778700] LNetError: 505467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6919.782870] LNetError: 505467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 6919.787443] LNet: Removed LNI 192.168.204.105@tcp30 [ 6920.800225] LNet: Removed LNI 192.168.204.105@tcp31 [ 6920.802676] LNet: Removed LNI 192.168.204.105@tcp32 [ 6920.804977] LNet: Removed LNI 192.168.204.105@tcp33 [ 6921.824083] LNet: Removed LNI 192.168.204.105@tcp34 [ 6922.848182] LNet: Removed LNI 192.168.204.105@tcp35 [ 6922.851176] LNet: Removed LNI 192.168.204.105@tcp36 [ 6923.872167] LNet: Removed LNI 192.168.204.105@tcp37 [ 6924.896110] LNet: Removed LNI 192.168.204.105@tcp38 [ 6925.920170] LNet: Removed LNI 192.168.204.105@tcp39 [ 6926.944145] LNet: Removed LNI 192.168.204.105@tcp40 [ 6927.968096] LNet: Removed LNI 192.168.204.105@tcp41 [ 6928.992093] LNet: Removed LNI 192.168.204.105@tcp42 [ 6928.993203] LNet: Removed LNI 192.168.204.105@tcp43 [ 6930.016180] LNet: Removed LNI 192.168.204.105@tcp44 [ 6930.018656] LNet: Removed LNI 192.168.204.105@tcp45 [ 6931.040176] LNet: Removed LNI 192.168.204.105@tcp46 [ 6932.064223] LNet: Removed LNI 192.168.204.105@tcp47 [ 6932.066650] LNet: Removed LNI 192.168.204.105@tcp48 [ 6932.068975] LNet: Removed LNI 192.168.204.105@tcp49 [ 6932.077741] LNet: Removed LNI 192.168.204.105@tcp50 [ 6932.456410] Key type .llcrypt unregistered [ 6932.458422] Key type ._llcrypt unregistered [ 6942.759382] Key type ._llcrypt registered [ 6942.761425] Key type .llcrypt registered [ 6942.827193] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 6943.291600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6943.340133] alg: No test for adler32 (adler32-zlib) [ 6944.193913] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 6944.278935] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 6945.856167] Key type lgssc registered [ 6946.453791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6950.476303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6950.487664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6951.740402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6953.652615] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6957.984248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6958.158645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6960.066941] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6962.734279] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6964.526414] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6967.767292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6967.967988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6970.307311] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6972.100667] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6980.939215] loop0: detected capacity change from 0 to 204800000 [ 6981.001565] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6981.004581] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6981.008251] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6981.032136] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6982.880210] loop0: detected capacity change from 0 to 204800000 [ 6982.943669] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6982.948471] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6982.952172] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6982.999481] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6984.537100] loop0: detected capacity change from 0 to 204800000 [ 6986.288042] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6986.308829] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6986.350196] Lustre: fs15246-OST0000: new disk, initializing [ 6986.351916] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 6986.369442] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6988.085468] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6991.676717] loop1: detected capacity change from 0 to 204800000 [ 6993.407462] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6993.438088] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6993.465033] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 6993.489402] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 6993.523277] Lustre: fs15246-MDT0000: new disk, initializing [ 6993.574671] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6993.582287] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6993.625307] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6994.657881] Lustre: 511069:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000006ccd9a46 x1852877242924288/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1767041504 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 6994.747714] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7003.626378] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7003.744145] LustreError: 511069:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7003.746855] LustreError: 511069:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 7003.749126] LustreError: 511069:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 7007.483779] LustreError: 512188:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 7007.511495] Lustre: server umount fs15246-MDT0000 complete [ 7008.741345] LustreError: 507286:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7008.745539] LustreError: 507286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7010.291132] LustreError: 512694:0:(obd_class.h:479:obd_check_dev()) Device 29 not setup [ 7010.292803] LustreError: 512694:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7010.305544] Lustre: server umount fs15246-OST0000 complete [ 7013.860595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7013.861447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7013.863653] Lustre: Skipped 1 previous similar message [ 7018.980985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7018.984890] Lustre: Skipped 2 previous similar messages [ 7019.444139] LustreError: 513195:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7019.447980] LustreError: 513195:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7019.509564] Lustre: server umount lustre-OST0000 complete [ 7024.100710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7024.102449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7024.108056] Lustre: Skipped 1 previous similar message [ 7027.534930] LustreError: 513397:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7027.538296] LustreError: 513397:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7027.575726] Lustre: server umount lustre-MDT0000 complete [ 7029.217907] LustreError: 506693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7029.227188] LustreError: 506693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7029.336796] LustreError: 506672:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767041523 with bad export cookie 1738906842281409237 [ 7029.343558] LustreError: 506672:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7029.343686] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7029.494191] Lustre: server umount lustre-MDT0001 complete [ 7033.689816] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 7035.132603] Key type lgssc unregistered [ 7035.288557] LNet: 514130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7035.293215] LNetError: 514130:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7035.301487] LNet: Removed LNI 192.168.204.105@tcp [ 7035.764211] Key type .llcrypt unregistered [ 7035.766259] Key type ._llcrypt unregistered [ 7044.985752] Key type ._llcrypt registered [ 7044.987798] Key type .llcrypt registered [ 7045.058819] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 7045.627730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7045.639101] alg: No test for adler32 (adler32-zlib) [ 7046.552510] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 7046.672407] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 7048.280200] Key type lgssc registered [ 7048.931962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7052.826535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7055.363351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7057.923349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7060.446369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7068.114354] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 7072.666269] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 7075.975721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7078.218464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7080.243993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7082.955941] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7085.505024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7085.523800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7085.527986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7086.647975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7086.664863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7086.710510] Lustre: lustre-MDT0000: new disk, initializing [ 7086.747466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7086.758223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7088.491836] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7093.817131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7093.851195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7093.907452] Lustre: 518537:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7093.948719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7093.952227] Lustre: Skipped 1 previous similar message [ 7094.001766] Lustre: lustre-MDT0001: new disk, initializing [ 7094.043313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7094.055945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7094.059365] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7095.332820] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7099.361655] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7101.117967] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7104.438844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7104.480484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7104.654514] Lustre: lustre-OST0000: new disk, initializing [ 7104.658359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7104.714742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7107.295350] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7111.421719] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7112.696191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7112.699385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7112.743800] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7113.269718] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7113.368259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7115.156902] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7115.276759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7117.797435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7117.799598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7117.810827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7117.814621] Lustre: Skipped 1 previous similar message [ 7122.291126] LustreError: 521478:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7122.355410] Lustre: server umount lustre-OST0000 complete [ 7124.961708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7124.966246] LustreError: Skipped 1 previous similar message [ 7124.968899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7124.975910] Lustre: Skipped 1 previous similar message [ 7124.979231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7128.034483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7128.034941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7130.322406] LustreError: 521679:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7130.326126] LustreError: 521679:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7130.362232] Lustre: server umount lustre-MDT0000 complete [ 7132.163440] LustreError: 519317:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767041626 with bad export cookie 7938209732592115313 [ 7132.167535] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7132.170467] LustreError: 519317:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7132.270660] LustreError: 521882:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7132.274071] LustreError: 521882:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7132.344511] Lustre: server umount lustre-MDT0001 complete [ 7142.967613] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 7146.080359] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 7149.293063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7151.441055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7153.402875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7155.455069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7156.010198] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6978 sec ========== 15:54:09 (1767041649) [ 7156.535588] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:54:09 (1767041649) === [ 7157.597685] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:54:10 (1767041650) === [ 7169.751463] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 7170.585097] Key type lgssc unregistered [ 7170.700324] LNet: 526730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7170.702503] LNetError: 526730:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7170.710313] LNet: Removed LNI 192.168.204.105@tcp [ 7170.962823] Key type .llcrypt unregistered [ 7170.963804] Key type ._llcrypt unregistered