[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 462262813 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003326] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008721] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011138] LSM: Security Framework initializing [ 0.012057] Yama: becoming mindful. [ 0.013043] SELinux: Initializing. [ 0.014064] *** VALIDATE selinux *** [ 0.023439] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029204] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031127] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033115] *** VALIDATE tmpfs *** [ 0.035266] *** VALIDATE proc *** [ 0.036246] *** VALIDATE cgroup *** [ 0.037009] *** VALIDATE cgroup2 *** [ 0.038282] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041029] Spectre V2 : User space: Vulnerable [ 0.042008] Speculative Store Bypass: Vulnerable [ 0.045123] debug: unmapping init [mem 0xffffffffab459000-0xffffffffab460fff] [ 0.047910] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048702] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049025] ... version: 2 [ 0.050014] ... bit width: 48 [ 0.051014] ... generic registers: 4 [ 0.052013] ... value mask: 0000ffffffffffff [ 0.053016] ... max period: 00007fffffffffff [ 0.054015] ... fixed-purpose events: 3 [ 0.055013] ... event mask: 000000070000000f [ 0.056324] rcu: Hierarchical SRCU implementation. [ 0.058434] smp: Bringing up secondary CPUs ... [ 0.059593] x86: Booting SMP configuration: [ 0.060023] .... node #0, CPUs: #1 #2 #3 [ 0.063719] smp: Brought up 1 node, 4 CPUs [ 0.065012] smpboot: Max logical packages: 1 [ 0.066019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.154883] node 0 deferred pages initialised in 86ms [ 0.156100] devtmpfs: initialized [ 0.157253] x86/mm: Memory block size: 128MB [ 0.161015] gcov: version magic: 0x41383552 [ 0.164089] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165112] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.166293] pinctrl core: initialized pinctrl subsystem [ 0.167188] [ 0.167789] ************************************************************* [ 0.168014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169012] ** ** [ 0.170011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171013] ** ** [ 0.172022] ** This means that this kernel is built to expose internal ** [ 0.173012] ** IOMMU data structures, which may compromise security on ** [ 0.174013] ** your system. ** [ 0.175012] ** ** [ 0.176011] ** If you see this message and you are not debugging the ** [ 0.177023] ** kernel, report this immediately to your vendor! ** [ 0.178013] ** ** [ 0.179013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180013] ************************************************************* [ 0.181582] NET: Registered protocol family 16 [ 0.182364] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.183058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.186013] cpuidle: using governor menu [ 0.187490] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.190427] PCI: Using configuration type 1 for base access [ 0.192134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.202043] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.203019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.204156] cryptd: max_cpu_qlen set to 1000 [ 0.207113] ACPI: Added _OSI(Module Device) [ 0.208011] ACPI: Added _OSI(Processor Device) [ 0.209011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.210045] ACPI: Added _OSI(Processor Aggregator Device) [ 0.214437] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.219661] ACPI: Interpreter enabled [ 0.221060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.222013] ACPI: Using IOAPIC for interrupt routing [ 0.224098] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.227453] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.237767] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.240047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.243023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.246077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.251408] acpiphp: Slot [2] registered [ 0.253120] acpiphp: Slot [5] registered [ 0.255127] acpiphp: Slot [6] registered [ 0.256154] acpiphp: Slot [7] registered [ 0.258096] acpiphp: Slot [8] registered [ 0.259099] acpiphp: Slot [9] registered [ 0.260054] acpiphp: Slot [10] registered [ 0.261010] acpiphp: Slot [3] registered [ 0.262089] acpiphp: Slot [4] registered [ 0.263092] acpiphp: Slot [11] registered [ 0.265107] acpiphp: Slot [12] registered [ 0.266121] acpiphp: Slot [13] registered [ 0.268129] acpiphp: Slot [14] registered [ 0.270100] acpiphp: Slot [15] registered [ 0.271101] acpiphp: Slot [16] registered [ 0.272108] acpiphp: Slot [17] registered [ 0.274303] acpiphp: Slot [18] registered [ 0.276115] acpiphp: Slot [19] registered [ 0.277089] acpiphp: Slot [20] registered [ 0.279101] acpiphp: Slot [21] registered [ 0.280079] acpiphp: Slot [22] registered [ 0.282093] acpiphp: Slot [23] registered [ 0.283079] acpiphp: Slot [24] registered [ 0.284077] acpiphp: Slot [25] registered [ 0.285091] acpiphp: Slot [26] registered [ 0.287101] acpiphp: Slot [27] registered [ 0.288099] acpiphp: Slot [28] registered [ 0.290104] acpiphp: Slot [29] registered [ 0.292125] acpiphp: Slot [30] registered [ 0.293112] acpiphp: Slot [31] registered [ 0.295091] PCI host bridge to bus 0000:00 [ 0.296017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.298023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.300023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.303025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.305026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.308034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.310205] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.312989] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.316412] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.325639] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.329902] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.332015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.334028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.337020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.339547] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.342562] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.344036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.346835] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.351013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.364015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.369013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.375139] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.385022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.401013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.432018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.442200] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.449020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.456018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.473019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.481478] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.487012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.491012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.506019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.518767] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.526018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.535019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.549018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.559138] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.566017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.570014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.590013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.597340] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.603015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.609018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.624024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.635584] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.637341] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.640415] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.642299] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.644214] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.648114] iommu: Default domain type: Passthrough [ 0.650471] SCSI subsystem initialized [ 0.652145] ACPI: bus type USB registered [ 0.653090] usbcore: registered new interface driver usbfs [ 0.655097] usbcore: registered new interface driver hub [ 0.657083] usbcore: registered new device driver usb [ 0.659182] pps_core: LinuxPPS API ver. 1 registered [ 0.661012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.664061] PTP clock support registered [ 0.666084] EDAC MC: Ver: 3.0.0 [ 0.667535] PCI: Using ACPI for IRQ routing [ 0.669856] NetLabel: Initializing [ 0.671013] NetLabel: domain hash size = 128 [ 0.673012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.675083] NetLabel: unlabeled traffic allowed by default [ 0.677137] vgaarb: loaded [ 0.679266] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.681016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.688051] clocksource: Switched to clocksource kvm-clock [ 0.796496] VFS: Disk quotas dquot_6.6.0 [ 0.798207] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.800643] *** VALIDATE ramfs *** [ 0.801933] *** VALIDATE hugetlbfs *** [ 0.803566] pnp: PnP ACPI init [ 0.806055] pnp: PnP ACPI: found 6 devices [ 0.821328] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.824829] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.826942] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.828967] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.831312] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.833527] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.836030] NET: Registered protocol family 2 [ 0.838503] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.843166] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.846578] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.851630] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.855224] TCP: Hash tables configured (established 65536 bind 65536) [ 0.858398] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.861218] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.864040] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.866845] NET: Registered protocol family 1 [ 0.868981] RPC: Registered named UNIX socket transport module. [ 0.870854] RPC: Registered udp transport module. [ 0.872379] RPC: Registered tcp transport module. [ 0.874429] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.876747] NET: Registered protocol family 44 [ 0.878224] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.879772] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.881574] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.883853] PCI: CLS 0 bytes, default 64 [ 0.886067] Unpacking initramfs... [ 2.224944] debug: unmapping init [mem 0xffff94103cc54000-0xffff94103ffbffff] [ 2.227526] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.229357] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.231071] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.693219] Initialise system trusted keyrings [ 2.694740] Key type blacklist registered [ 2.696653] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.706735] zbud: loaded [ 2.710208] *** VALIDATE nfs *** [ 2.711415] *** VALIDATE nfs4 *** [ 2.712827] pstore: using deflate compression [ 2.716874] Platform Keyring initialized [ 2.813212] NET: Registered protocol family 38 [ 2.814420] Key type asymmetric registered [ 2.815415] Asymmetric key parser 'x509' registered [ 2.816615] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.818495] io scheduler mq-deadline registered [ 2.819539] io scheduler kyber registered [ 2.820874] io scheduler bfq registered [ 2.822650] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.824830] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.826809] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.829058] ACPI: Power Button [PWRF] [ 2.833065] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.838178] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.853087] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.857436] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.866346] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.891243] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.917868] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.921811] Non-volatile memory driver v1.3 [ 2.923080] Linux agpgart interface v0.103 [ 2.946531] virtio_blk virtio1: [vda] 134024 512-byte logical blocks (68.6 MB/65.4 MiB) [ 2.948437] vda: detected capacity change from 0 to 68620288 [ 2.959593] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.961729] vdb: detected capacity change from 0 to 1073741824 [ 2.976149] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.979518] vdc: detected capacity change from 0 to 2621440000 [ 2.994045] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.997254] vdd: detected capacity change from 0 to 2621440000 [ 3.009971] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.012423] vde: detected capacity change from 0 to 4294967296 [ 3.023938] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.025916] vdf: detected capacity change from 0 to 4294967296 [ 3.030887] libphy: Fixed MDIO Bus: probed [ 3.038210] usbcore: registered new interface driver usbserial_generic [ 3.040084] usbserial: USB Serial support registered for generic [ 3.041681] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.044916] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.046400] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.048237] mousedev: PS/2 mouse device common for all mice [ 3.050201] rtc_cmos 00:05: RTC can wake from S4 [ 3.050932] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.055121] rtc_cmos 00:05: registered as rtc0 [ 3.058043] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.060903] intel_pstate: CPU model not supported [ 3.063539] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.066285] hid: raw HID events driver (C) Jiri Kosina [ 3.068368] usbcore: registered new interface driver usbhid [ 3.071021] usbhid: USB HID core driver [ 3.072841] drop_monitor: Initializing network drop monitor service [ 3.072855] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.075025] Initializing XFRM netlink socket [ 3.079039] NET: Registered protocol family 10 [ 3.081199] Segment Routing with IPv6 [ 3.082714] NET: Registered protocol family 17 [ 3.086379] mpls_gso: MPLS GSO support [ 3.090971] RAS: Correctable Errors collector initialized. [ 3.092580] AVX version of gcm_enc/dec engaged. [ 3.094334] AES CTR mode by8 optimization enabled [ 3.154515] sched_clock: Marking stable (3154492001, 0)->(4051715708, -897223707) [ 3.158153] registered taskstats version 1 [ 3.159851] Loading compiled-in X.509 certificates [ 3.161818] zswap: loaded using pool lzo/zbud [ 3.183650] Key type big_key registered [ 3.194531] Key type encrypted registered [ 3.196034] ima: No TPM chip found, activating TPM-bypass! [ 3.198119] ima: Allocated hash algorithm: sha1 [ 3.200075] ima: No architecture policies found [ 3.202103] evm: Initialising EVM extended attributes: [ 3.203850] evm: security.selinux [ 3.205020] evm: security.ima [ 3.205865] evm: security.capability [ 3.207055] evm: HMAC attrs: 0x1 [ 3.209849] rtc_cmos 00:05: setting system clock to 2026-01-02 03:10:22 UTC (1767323422) [ 3.216590] debug: unmapping init [mem 0xffffffffac403000-0xffffffffac5fffff] [ 3.219533] debug: unmapping init [mem 0xffffffffab182000-0xffffffffab458fff] [ 3.229074] Write protecting the kernel read-only data: 28672k [ 3.232600] debug: unmapping init [mem 0xffffffffa9803000-0xffffffffa99fffff] [ 3.235268] debug: unmapping init [mem 0xffffffffaa114000-0xffffffffaa1fffff] [ 3.271311] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.279283] systemd[1]: Detected virtualization kvm. [ 3.281096] systemd[1]: Detected architecture x86-64. [ 3.282748] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.311043] systemd[1]: No hostname configured. [ 3.312625] systemd[1]: Set hostname to . [ 3.314386] random: systemd: uninitialized urandom read (16 bytes read) [ 3.316586] systemd[1]: Initializing machine ID from random generator. [ 3.365730] random: ln: uninitialized urandom read (6 bytes read) [ 3.439547] random: systemd: uninitialized urandom read (16 bytes read) [ 3.442345] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.446207] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.454780] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.987093] device-mapper: uevent: version 1.0.3 [ 3.989080] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.664450] random: fast init done [ 4.664493] virtio_net virtio0 ens2: renamed from eth0 [ 4.710859] scsi host0: ata_piix [ 4.713804] scsi host1: ata_piix [ 4.715827] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.718021] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.182702] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.715570] random: crng init done [ 9.716826] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.147205] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.313380] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.555469] SELinux: Disabled at runtime. [ 11.613715] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.622830] systemd[1]: Detected virtualization kvm. [ 11.625059] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.072680] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.076071] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.080233] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.084291] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.088827] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.096176] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.114354] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ 12.158934] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.552099] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.817081] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.896718] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.067383] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.075347] EDAC sbridge: Ver: 1.1.2 [ 14.725959] Key type dns_resolver registered [ 15.036517] NFS: Registering the id_resolver key type [ 15.037753] Key type id_resolver registered [ 15.038669] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg423-server login: [ 45.871508] libcfs: loading out-of-tree module taints kernel. [ 45.893059] Key type ._llcrypt registered [ 45.894600] Key type .llcrypt registered [ 45.945321] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 100.828108] hrtimer: interrupt took 8099900 ns [ 101.949474] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 103.448618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 103.480249] alg: No test for adler32 (adler32-zlib) [ 104.877692] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 105.774067] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 107.616171] Key type lgssc registered [ 109.062365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 130.365892] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 130.737804] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 131.176965] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 131.880866] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 132.044906] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 186.961820] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 202.713546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 202.762547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 202.782671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.974808] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 203.997628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 204.052990] Lustre: lustre-MDT0000: new disk, initializing [ 204.118434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.139831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 207.614227] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.380101] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 227.407377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 227.465259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 227.518732] Lustre: 14422:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 227.552581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 227.557503] Lustre: Skipped 1 previous similar message [ 227.630624] Lustre: lustre-MDT0001: new disk, initializing [ 227.679255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 227.700093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 227.718097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 231.440888] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.516114] Lustre: 16478:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 251.018535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 251.072570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 251.279525] Lustre: lustre-OST0000: new disk, initializing [ 251.283390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 251.336046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 251.534244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 251.545355] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 251.614553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 256.132829] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 272.280761] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 272.360614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 272.434747] Lustre: lustre-OST0001: new disk, initializing [ 272.439339] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 272.489836] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 276.540306] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 281.635108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 281.639169] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 281.667493] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 286.383707] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 295.828959] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 304.804785] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing check_logdir /tmp/testlogs/ [ 307.585834] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing yml_node [ 312.892316] Lustre: DEBUG MARKER: Client: 2.17.0.RC4 [ 314.389989] Lustre: DEBUG MARKER: MDS: 2.17.0.RC4 [ 315.992701] Lustre: DEBUG MARKER: OSS: 2.17.0.RC4 [ 317.032966] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jan 1 22:15:36 EST 2026 [ 333.898593] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 334.776255] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 348.130419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.132355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 348.141975] Lustre: Skipped 3 previous similar messages [ 348.148736] Lustre: Skipped 3 previous similar messages [ 353.249278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 353.251910] Lustre: Skipped 1 previous similar message [ 354.086094] LustreError: 21868:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 354.215739] Lustre: server umount lustre-MDT0000 complete [ 359.622700] LustreError: 14414:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323778 with bad export cookie 13331580253024831852 [ 359.625051] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 359.628864] LustreError: 14414:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 363.488971] LustreError: 14429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 363.490066] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 363.492648] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 363.497712] LustreError: 14429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 363.529654] Lustre: Skipped 1 previous similar message [ 364.513567] LustreError: 19662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 364.526115] LustreError: 19662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 365.782117] LustreError: 22319:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 365.785293] LustreError: 22319:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 365.854578] Lustre: server umount lustre-MDT0001 complete [ 371.910399] LustreError: 22915:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 371.913633] LustreError: 22915:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 371.942535] Lustre: server umount lustre-OST0000 complete [ 376.822276] LustreError: 23364:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 376.828134] LustreError: 23364:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 376.902355] Lustre: server umount lustre-OST0001 complete [ 385.139960] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 426.792583] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 438.269129] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 438.418670] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 439.417638] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 441.628963] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 452.742663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 452.792583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 452.944441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 452.971936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 453.022601] Lustre: lustre-MDT0000: new disk, initializing [ 453.074482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 453.084329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 455.683288] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 466.316260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 466.364533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 466.428731] Lustre: 30562:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 466.465227] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 466.467244] Lustre: Skipped 1 previous similar message [ 466.517540] Lustre: lustre-MDT0001: new disk, initializing [ 466.591917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 466.620987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 466.625563] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 469.248301] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 474.137041] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 476.561480] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 485.381170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 485.427372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 485.577850] Lustre: lustre-OST0000: new disk, initializing [ 485.583787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 485.645322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 487.402804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 487.411510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 487.487910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 489.662462] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 495.198504] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 497.861630] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 497.972670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 500.560186] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 500.747032] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 502.753448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 502.755728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.757771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 502.760237] Lustre: Skipped 1 previous similar message [ 502.766135] Lustre: Skipped 3 previous similar messages [ 507.876552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 507.882342] Lustre: Skipped 1 previous similar message [ 508.092301] LustreError: 34191:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 508.096688] LustreError: 34191:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 508.158239] Lustre: server umount lustre-OST0000 complete [ 512.997137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 513.003042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 513.005299] LustreError: Skipped 1 previous similar message [ 513.025346] Lustre: Skipped 1 previous similar message [ 517.306827] LustreError: 34440:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 517.312641] LustreError: 34440:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 517.375491] Lustre: server umount lustre-MDT0000 complete [ 518.115281] LustreError: 30573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.131926] LustreError: 30573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 520.662552] LustreError: 30552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323939 with bad export cookie 13331580253024833406 [ 520.667886] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 520.674802] LustreError: 30552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 520.856496] Lustre: server umount lustre-MDT0001 complete [ 523.768348] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 22:19:02 (1767323942) [ 526.253834] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 527.589731] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 22:19:06 (1767323946) [ 534.196886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.443428] LustreError: 35600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.507580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 537.100781] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 539.619213] LustreError: 35600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.695323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 547.231265] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 550.840314] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 553.260504] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 560.221293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 566.048646] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 570.285331] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 589.283259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.285834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 589.286639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 589.294118] Lustre: Skipped 1 previous similar message [ 589.303910] Lustre: Skipped 3 previous similar messages [ 592.821417] LustreError: 38251:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 592.825522] LustreError: 38251:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 592.893240] Lustre: server umount lustre-OST0000 complete [ 599.527301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 602.772458] Lustre: server umount lustre-MDT0000 complete [ 604.641298] LustreError: 35599:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 604.663478] LustreError: 35599:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 606.352103] LustreError: 35580:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324025 with bad export cookie 13331580253024834281 [ 606.357605] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 606.360054] LustreError: 35580:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 606.575920] Lustre: server umount lustre-MDT0001 complete [ 616.710786] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 618.567305] Key type lgssc unregistered [ 618.745703] LNet: 39427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.750429] LNetError: 39427:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 618.762240] LNet: Removed LNI 192.168.204.123@tcp [ 619.393157] Key type .llcrypt unregistered [ 619.395528] Key type ._llcrypt unregistered [ 671.995179] Key type ._llcrypt registered [ 671.996542] Key type .llcrypt registered [ 672.066875] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 672.733041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 672.746307] alg: No test for adler32 (adler32-zlib) [ 673.689571] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 673.823015] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 675.464206] Key type lgssc registered [ 676.092585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 681.738512] loop: module loaded [ 681.797535] loop0: detected capacity change from 0 to 204800000 [ 681.947177] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 681.951342] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 682.106113] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 682.140820] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 686.017197] loop0: detected capacity change from 0 to 204800000 [ 686.131738] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 686.139457] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 686.145524] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 686.260497] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 691.714160] loop0: detected capacity change from 0 to 204800000 [ 694.791720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 694.838454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 694.854150] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.006564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 696.059781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 696.126995] Lustre: lustre-MDT0000: new disk, initializing [ 696.193099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 696.207906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 698.657424] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 707.249449] loop1: detected capacity change from 0 to 204800000 [ 710.687583] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 710.736385] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 710.864848] Lustre: lustre-OST07c6: new disk, initializing [ 710.869376] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 710.872959] Lustre: Skipped 1 previous similar message [ 710.915900] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 712.805333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 712.812803] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 712.844802] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 713.908192] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 716.869777] Lustre: 45123:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 716.873884] Lustre: 45123:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 716.878445] Lustre: 45123:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 716.881672] Lustre: 45123:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 716.887942] Lustre: 45123:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 716.894261] Lustre: 45123:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 734.551856] Lustre: 45123:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 734.561288] Lustre: 45123:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 734.565026] Lustre: 45123:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 734.568315] Lustre: 45123:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 734.571320] Lustre: 45123:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 734.574462] Lustre: 45123:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 738.729417] Lustre: 45124:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 738.738309] Lustre: 45124:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 738.746322] Lustre: 45124:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 738.754465] Lustre: 45124:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 738.766412] Lustre: 45124:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 738.780398] Lustre: 45124:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 738.787896] Lustre: 45124:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 738.790825] Lustre: 45124:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 738.794594] Lustre: 45124:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 738.801776] Lustre: 45124:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 738.813930] Lustre: 45124:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 738.825837] Lustre: 45124:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.996524] Lustre: 45124:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 742.000539] Lustre: 45124:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 742.005208] Lustre: 45124:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 742.009227] Lustre: 45124:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 742.015853] Lustre: 45124:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 742.021096] Lustre: 45124:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 742.027974] Lustre: 45124:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 742.034597] Lustre: 45124:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 742.040536] Lustre: 45124:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 742.046207] Lustre: 45124:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 742.052658] Lustre: 45124:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 742.058328] Lustre: 45124:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.163841] Lustre: 45123:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 746.168621] Lustre: 45123:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 746.173264] Lustre: 45123:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 746.177808] Lustre: 45123:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.182035] Lustre: 45123:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 746.186760] Lustre: 45123:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.190500] Lustre: 45123:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 746.197426] Lustre: 45123:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.201746] Lustre: 45123:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 746.205761] Lustre: 45123:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.210115] Lustre: 45123:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 746.215246] Lustre: 45123:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.711268] Lustre: 45121:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 756.719676] Lustre: 45121:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 5 previous similar messages [ 756.730607] Lustre: 45121:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 756.739896] Lustre: 45121:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 756.745700] Lustre: 45121:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 756.750916] Lustre: 45121:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 756.757815] Lustre: 45121:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 756.764720] Lustre: 45121:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 756.770710] Lustre: 45121:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 756.777021] Lustre: 45121:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 756.782344] Lustre: 45121:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 756.788444] Lustre: 45121:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 804.845426] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 804.850811] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 804.852843] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 809.954906] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 815.073799] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 816.096144] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 816.129094] LustreError: 47599:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 816.208236] Lustre: server umount lustre-OST07c6 complete [ 821.949953] LustreError: 48102:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 821.955874] LustreError: 48102:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 822.086913] Lustre: server umount lustre-MDT0000 complete [ 833.854705] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 835.694314] Key type lgssc unregistered [ 835.892517] LNet: 49078:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.897787] LNetError: 49078:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.909664] LNet: Removed LNI 192.168.204.123@tcp [ 836.440762] Key type .llcrypt unregistered [ 836.442469] Key type ._llcrypt unregistered [ 885.426225] Key type ._llcrypt registered [ 885.428560] Key type .llcrypt registered [ 885.497217] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 886.160033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 886.182879] alg: No test for adler32 (adler32-zlib) [ 887.148934] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 887.267457] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 888.880259] Key type lgssc registered [ 889.637962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 894.239305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 894.248259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 895.538130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 897.935368] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 905.224035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 905.431463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 907.893911] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 911.169917] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 913.494611] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 920.227519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 920.468045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 921.524515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 924.631535] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 927.888526] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 946.149715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 946.154094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 946.156659] Lustre: Skipped 1 previous similar message [ 946.161630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 946.165891] Lustre: Skipped 1 previous similar message [ 948.474090] LustreError: 56217:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 948.538780] Lustre: server umount lustre-OST0000 complete [ 951.265549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 951.269071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 951.275343] Lustre: Skipped 1 previous similar message [ 951.282571] Lustre: Skipped 1 previous similar message [ 956.385315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 956.390080] Lustre: Skipped 1 previous similar message [ 957.365919] LustreError: 56467:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 957.374427] LustreError: 56467:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 957.428746] Lustre: server umount lustre-MDT0000 complete [ 960.097284] LustreError: 54297:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324379 with bad export cookie 11756385817588630976 [ 960.100299] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 960.103461] LustreError: 54297:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 960.190813] LustreError: 56718:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 960.194379] LustreError: 56718:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 960.284865] Lustre: server umount lustre-MDT0001 complete [ 967.501546] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 969.107953] Key type lgssc unregistered [ 969.302563] LNet: 57395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.309634] LNetError: 57395:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.325198] LNet: Removed LNI 192.168.204.123@tcp [ 969.830029] Key type .llcrypt unregistered [ 969.831362] Key type ._llcrypt unregistered [ 1011.322235] Key type ._llcrypt registered [ 1011.323571] Key type .llcrypt registered [ 1011.366455] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1011.835333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1011.846761] alg: No test for adler32 (adler32-zlib) [ 1012.734399] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 1012.845990] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1014.448224] Key type lgssc registered [ 1014.901292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1018.568764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1018.581864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1019.781932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.602754] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1027.433923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1027.650463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1029.902487] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1032.911113] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1034.708810] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1039.728331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1039.958622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1042.738139] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1045.097498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1045.174459] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1055.713337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1055.717265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1055.724493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1057.763174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1057.763396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1057.765601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1057.766448] Lustre: Skipped 1 previous similar message [ 1061.614840] LustreError: 64330:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1061.685931] Lustre: server umount lustre-OST0000 complete [ 1068.003300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.005036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1068.007622] Lustre: Skipped 1 previous similar message [ 1070.168728] LustreError: 64580:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1070.172373] LustreError: 64580:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1070.252548] Lustre: server umount lustre-MDT0000 complete [ 1072.427403] LustreError: 61681:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324491 with bad export cookie 12515716651162786128 [ 1072.433027] LustreError: 61681:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1072.433856] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1072.538982] LustreError: 64829:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1072.546497] LustreError: 64829:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1072.618251] Lustre: server umount lustre-MDT0001 complete [ 1079.458337] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1080.747319] Key type lgssc unregistered [ 1080.934621] LNet: 65507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1080.938385] LNetError: 65507:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1080.950658] LNet: Removed LNI 192.168.204.123@tcp [ 1081.371278] Key type .llcrypt unregistered [ 1081.374332] Key type ._llcrypt unregistered [ 1123.450352] Key type ._llcrypt registered [ 1123.452315] Key type .llcrypt registered [ 1123.507763] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1124.103569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1124.111428] alg: No test for adler32 (adler32-zlib) [ 1125.026740] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 1125.139148] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1126.744164] Key type lgssc registered [ 1127.187725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1130.775784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1130.788076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1132.016118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1133.689975] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1139.065682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1139.212396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1140.786115] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1143.046213] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1144.799159] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1149.873664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1150.075404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1152.648573] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1154.984151] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1157.222115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1169.890130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.892594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1169.894270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1169.901695] Lustre: Skipped 1 previous similar message [ 1172.694642] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 1172.698050] Lustre: Skipped 2 previous similar messages [ 1175.011179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1175.014017] Lustre: Skipped 2 previous similar messages [ 1175.301035] LustreError: 72655:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1175.344841] Lustre: server umount lustre-MDT0000 complete [ 1177.679385] LustreError: 69936:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324596 with bad export cookie 9581393803283239948 [ 1177.682465] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1177.683970] LustreError: 69936:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1177.802545] LustreError: 72497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1177.810913] LustreError: 72497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1177.811124] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.23@tcp (stopping) [ 1180.129231] LustreError: 69953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1180.129473] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.145589] Lustre: Skipped 1 previous similar message [ 1182.177341] LustreError: 72497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1182.177580] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1182.188059] Lustre: Skipped 1 previous similar message [ 1183.805595] LustreError: 72906:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1183.807250] LustreError: 72906:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1183.859629] Lustre: server umount lustre-MDT0001 complete [ 1202.610918] LustreError: 73129:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1202.614152] LustreError: 73129:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1202.678754] Lustre: server umount lustre-OST0000 complete [ 1211.587630] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1212.880881] Key type lgssc unregistered [ 1213.024614] LNet: 74205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1213.028305] LNetError: 74205:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1213.036315] LNet: Removed LNI 192.168.204.123@tcp [ 1213.402879] Key type .llcrypt unregistered [ 1213.404043] Key type ._llcrypt unregistered [ 1250.220235] Key type ._llcrypt registered [ 1250.221876] Key type .llcrypt registered [ 1250.276278] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1250.612158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1250.626942] alg: No test for adler32 (adler32-zlib) [ 1251.519350] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 1251.615115] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1253.216143] Key type lgssc registered [ 1253.655950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1256.850434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1256.858526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1258.036588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1259.509779] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1264.511416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1264.664490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1266.171400] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1268.566277] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1270.327982] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1275.271360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1275.422032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1277.982611] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1280.676042] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1283.624963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1298.646804] Lustre: lustre-OST0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 1298.915094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1298.915309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1298.922529] LustreError: Skipped 1 previous similar message [ 1303.756061] Lustre: lustre-OST0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 1303.761222] Lustre: Skipped 2 previous similar messages [ 1308.875549] Lustre: lustre-OST0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 1308.877602] Lustre: Skipped 2 previous similar messages [ 1309.152127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1309.177358] LustreError: 81305:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1309.217406] Lustre: server umount lustre-OST0000 complete [ 1323.186649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.187611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1323.191320] Lustre: Skipped 2 previous similar messages [ 1323.196092] Lustre: Skipped 3 previous similar messages [ 1328.612399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1328.616179] Lustre: Skipped 1 previous similar message [ 1329.409695] LustreError: 81731:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1329.420226] LustreError: 81731:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1329.473469] Lustre: server umount lustre-MDT0000 complete [ 1331.300358] LustreError: 81732:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324750 with bad export cookie 2875341173077403863 [ 1331.303288] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1331.304505] LustreError: 81732:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1337.420248] LustreError: 81981:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1337.423716] LustreError: 81981:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1337.489348] Lustre: server umount lustre-MDT0001 complete [ 1342.969408] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1344.252805] Key type lgssc unregistered [ 1344.416455] LNet: 82657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1344.420686] LNetError: 82657:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1344.429581] LNet: Removed LNI 192.168.204.123@tcp [ 1344.838393] Key type .llcrypt unregistered [ 1344.845559] Key type ._llcrypt unregistered [ 1381.499832] Key type ._llcrypt registered [ 1381.501291] Key type .llcrypt registered [ 1381.549152] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1381.899185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1381.939367] alg: No test for adler32 (adler32-zlib) [ 1382.824722] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 1382.920682] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1384.529202] Key type lgssc registered [ 1385.229674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1389.647422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1389.662313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.935401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1393.224758] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1400.091831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1400.275356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1402.424281] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1405.506371] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1407.477573] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1413.377528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1413.569853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1416.678665] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1416.683303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1419.799869] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1427.921507] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1429.441124] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1440.402876] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-01-0122h34m19s [ 1441.153575] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1442.516875] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 1443.529966] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 1445.346524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.352962] Lustre: Skipped 2 previous similar messages [ 1445.353617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1445.360097] Lustre: Skipped 2 previous similar messages [ 1448.518205] LustreError: 90110:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1448.563469] Lustre: server umount lustre-MDT0000 complete [ 1449.680627] LustreError: 87109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.688587] LustreError: 87109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1450.394653] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1452.515785] LustreError: 87109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1452.524708] LustreError: 87109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1454.729216] LustreError: 87107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1454.935224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1454.995234] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1455.106489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1455.139524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1456.169151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1456.776141] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1458.254829] Lustre: DEBUG MARKER: Wait for df (44194) ... [ 1458.866386] Lustre: DEBUG MARKER: done [ 1459.248133] LustreError: 87092:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.23@tcp arrived at 1767324878 with bad export cookie 6876284050458482291 [ 1459.252791] LustreError: 87092:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1460.193313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1460.196165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1460.201250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1460.203793] Lustre: Skipped 3 previous similar messages [ 1460.705106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1462.064788] LustreError: 91349:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1462.066904] LustreError: 91349:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1462.107025] Lustre: server umount lustre-OST0000 complete [ 1466.338053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.338645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1466.339208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1466.344529] Lustre: Skipped 2 previous similar messages [ 1466.351841] Lustre: Skipped 4 previous similar messages [ 1477.603213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1477.606326] Lustre: Skipped 4 previous similar messages [ 1478.112211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1478.193581] LustreError: 91601:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1478.201026] LustreError: 91601:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1478.236187] Lustre: server umount lustre-MDT0000 complete [ 1478.627112] LustreError: 87112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1478.643233] LustreError: 87112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1480.577410] LustreError: 89791:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324899 with bad export cookie 6876284050458482515 [ 1480.579609] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1480.585615] LustreError: 89791:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1480.669258] LustreError: 91851:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1480.673170] LustreError: 91851:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1480.739734] Lustre: server umount lustre-MDT0001 complete [ 1486.966399] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1488.171499] Key type lgssc unregistered [ 1488.308770] LNet: 92526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.311525] LNetError: 92526:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1488.321030] LNet: Removed LNI 192.168.204.123@tcp [ 1488.685996] Key type .llcrypt unregistered [ 1488.687561] Key type ._llcrypt unregistered [ 1516.453675] Key type ._llcrypt registered [ 1516.455209] Key type .llcrypt registered [ 1516.512174] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 1553.108290] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1553.695767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1553.704553] alg: No test for adler32 (adler32-zlib) [ 1554.630298] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 1554.744993] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1556.345057] Key type lgssc registered [ 1556.940495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1565.218065] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1565.680952] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1566.405893] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1567.946469] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1575.947848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1575.993275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1576.000793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1577.122428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1577.138647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1577.184420] Lustre: lustre-MDT0000: new disk, initializing [ 1577.229158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1577.241978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1579.249599] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1588.084248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1588.133021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1588.189605] Lustre: 102345:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1588.221378] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1588.223753] Lustre: Skipped 1 previous similar message [ 1588.264396] Lustre: lustre-MDT0001: new disk, initializing [ 1588.319685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1588.338074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1588.344821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1590.271670] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1594.460658] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1596.457740] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1604.277570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1604.329252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1604.515599] Lustre: lustre-OST0000: new disk, initializing [ 1604.520493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1604.581412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1607.952639] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1610.266810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1610.273059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1610.328902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1612.283351] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1614.114470] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1614.213100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1616.299519] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1616.408214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1620.451140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1620.452265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1620.455514] LustreError: Skipped 1 previous similar message [ 1620.456970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1620.463637] Lustre: Skipped 1 previous similar message [ 1623.491399] LustreError: 105975:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1623.552717] Lustre: server umount lustre-OST0000 complete [ 1629.153288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1629.156813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1629.162455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1629.164552] Lustre: Skipped 1 previous similar message [ 1630.690130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1630.693556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1632.206938] LustreError: 106225:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1632.214083] LustreError: 106225:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1632.288795] Lustre: server umount lustre-MDT0000 complete [ 1634.685357] LustreError: 102336:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325053 with bad export cookie 5408879605965257275 [ 1634.689231] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1634.692913] LustreError: 102336:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1634.780560] LustreError: 106474:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1634.783108] LustreError: 106474:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1634.845166] Lustre: server umount lustre-MDT0001 complete [ 1639.127047] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 22:37:38 (1767325058) [ 1643.817690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1644.110581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.907550] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1649.122824] LustreError: 107336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1649.128915] LustreError: 107336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1651.552363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1651.724129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1653.816301] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1656.236972] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1657.729545] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1662.059838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1662.207272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1664.533536] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1666.607740] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1679.235535] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-01-0122h38m18s [ 1679.839856] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1682.528700] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1683.513710] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1683.531387] LustreError: 107331:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1703.638234] Lustre: lustre-MDT0000: Client 5b371baa-ab45-41b4-a2c6-41cb4a9e249c (at 192.168.204.23@tcp) reconnecting [ 1723.584121] LustreError: 107331:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1724.471667] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1725.377140] Lustre: DEBUG MARKER: done [ 1727.970761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1727.970797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1727.971924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1727.971929] Lustre: Skipped 1 previous similar message [ 1727.980128] Lustre: Skipped 1 previous similar message [ 1733.090739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1733.093164] Lustre: Skipped 2 previous similar messages [ 1733.166916] LustreError: 110747:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1733.170064] LustreError: 110747:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1733.206454] Lustre: server umount lustre-OST0000 complete [ 1738.210311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1740.989539] LustreError: 110996:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1740.991726] LustreError: 110996:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1741.019293] Lustre: server umount lustre-MDT0000 complete [ 1742.914630] LustreError: 110134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325162 with bad export cookie 5408879605965258157 [ 1742.918157] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1742.919846] LustreError: 110134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1743.071738] Lustre: server umount lustre-MDT0001 complete [ 1748.368152] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1749.661963] Key type lgssc unregistered [ 1749.840486] LNet: 111920:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.843789] LNetError: 111920:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.862322] LNet: Removed LNI 192.168.204.123@tcp [ 1750.203592] Key type .llcrypt unregistered [ 1750.205152] Key type ._llcrypt unregistered [ 1775.284448] Key type ._llcrypt registered [ 1775.285602] Key type .llcrypt registered [ 1775.331942] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 1805.625683] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1806.214304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1806.238525] alg: No test for adler32 (adler32-zlib) [ 1807.118470] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 1807.218196] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1808.808264] Key type lgssc registered [ 1809.312029] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1816.853245] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1817.094169] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1817.883252] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1819.031585] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1825.916866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1825.940197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1825.949271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1827.054500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1827.068047] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1827.107308] Lustre: lustre-MDT0000: new disk, initializing [ 1827.138606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1827.148305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1828.798941] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1836.957769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1836.988502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1837.038547] Lustre: 121740:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1837.072848] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1837.075899] Lustre: Skipped 1 previous similar message [ 1837.115458] Lustre: lustre-MDT0001: new disk, initializing [ 1837.168015] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1837.186929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1837.192237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1839.240384] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1843.301106] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1844.765788] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1850.279036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1850.306025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1850.412421] Lustre: lustre-OST0000: new disk, initializing [ 1850.415322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1850.446807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1851.141954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1851.145534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1851.193311] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1852.660744] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1856.234137] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1858.088242] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1858.202436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1859.866366] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1859.969026] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1861.601928] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1861.602147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.607851] LustreError: Skipped 1 previous similar message [ 1861.609744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1861.614804] Lustre: Skipped 1 previous similar message [ 1866.727411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1866.735640] Lustre: Skipped 2 previous similar messages [ 1866.928234] LustreError: 125368:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1866.987297] Lustre: server umount lustre-OST0000 complete [ 1871.841532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1871.843782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1871.850929] Lustre: Skipped 1 previous similar message [ 1871.862965] Lustre: Skipped 1 previous similar message [ 1875.268674] LustreError: 125617:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1875.271369] LustreError: 125617:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1875.316865] Lustre: server umount lustre-MDT0000 complete [ 1877.428190] LustreError: 121730:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325296 with bad export cookie 16844581468692721861 [ 1877.431746] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1877.432434] LustreError: 121730:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1877.503861] LustreError: 125867:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1877.506431] LustreError: 125867:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1877.555447] Lustre: server umount lustre-MDT0001 complete [ 1880.914713] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 22:41:40 (1767325300) [ 1881.604347] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1882.207307] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 22:41:41 (1767325301) [ 1907.449404] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1909.084592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1909.237906] LustreError: 129846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1909.245029] LustreError: 129846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1909.256091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1909.627498] Lustre: Failing over lustre-MDT0000 [ 1909.629930] LustreError: 129874:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1909.635527] LustreError: 129874:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1909.641518] LustreError: 129874:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1909.709952] LustreError: 129925:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1909.712490] LustreError: 129925:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1909.762319] Lustre: server umount lustre-MDT0000 complete [ 1912.217264] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 22:42:11 (1767325331) [ 1916.209867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1916.382889] LustreError: 130685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1916.413158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1917.925112] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1921.505217] LustreError: 130685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1922.818670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1924.423876] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1926.554558] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1927.929900] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1931.982354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1932.106281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.109726] Lustre: Skipped 1 previous similar message [ 1934.288977] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1936.366862] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1943.915279] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1948.641089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1948.643848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.650788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1951.714426] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.724255] Lustre: Skipped 1 previous similar message [ 1953.991288] LustreError: 133462:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1953.993343] LustreError: 133462:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1954.023143] Lustre: server umount lustre-MDT0000 complete [ 1956.111663] LustreError: 130665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325375 with bad export cookie 16844581468692723016 [ 1956.114601] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1956.116545] LustreError: 130665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1956.245565] Lustre: server umount lustre-MDT0001 complete [ 1957.750151] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1960.080562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1966.336241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.152766] Lustre: 118218:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767325376/real 1767325376] req@00000000f5523a0d x1853174816240640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767325392 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1973.173049] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1978.336307] Lustre: 118219:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767325381/real 1767325381] req@000000005a94d1ad x1853174816240768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767325397 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1981.408808] LustreError: 118216:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000040bccdfb x1853174816241664/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1981.570865] LustreError: 132677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1981.611867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1981.650460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1982.625104] Lustre: 118220:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767325385/real 1767325385] req@0000000048a45d2a x1853174816241024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767325401 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1983.257861] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1986.728436] Lustre: 118217:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767325390/real 1767325390] req@0000000019bb4a61 x1853174816241152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767325406 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1988.912259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1990.781788] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1992.168796] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1992.761273] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1994.184341] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1997.284326] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1997.285571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1997.288381] Lustre: Skipped 1 previous similar message [ 1997.291781] Lustre: Skipped 5 previous similar messages [ 2002.631684] LustreError: 136503:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2002.634812] LustreError: 136503:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2002.680381] Lustre: server umount lustre-MDT0000 complete [ 2004.591210] LustreError: 131595:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325423 with bad export cookie 16844581468692724920 [ 2004.594280] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2004.596102] LustreError: 131595:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2004.718794] Lustre: server umount lustre-MDT0001 complete [ 2007.743183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2013.504324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2014.624104] LustreError: 137882:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2014.627398] LustreError: 137882:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@000000008ed237ac x1853174816260992/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767325433 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2014.634876] LustreError: 137882:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2014.689216] LustreError: 118216:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a0c0c478 x1853174816261632/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2014.818164] LustreError: 135773:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2014.827621] LustreError: 135773:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2014.854496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.857309] Lustre: Skipped 1 previous similar message [ 2014.884325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2016.371913] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2020.327287] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.338143] Lustre: Skipped 1 previous similar message [ 2022.310844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2023.524654] Lustre: 118219:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767325426/real 1767325426] req@000000001b36026a x1853174816261248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767325442 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2023.533494] Lustre: 118219:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2024.330890] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2026.842361] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2027.687143] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2027.690832] Lustre: Skipped 1 previous similar message [ 2028.408958] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2033.123954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2033.125469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2033.130924] Lustre: Skipped 5 previous similar messages [ 2037.126960] LustreError: 139535:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2037.129074] LustreError: 139535:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2038.245485] LustreError: 137893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2038.251281] LustreError: 137893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2039.226681] Lustre: server umount lustre-MDT0000 complete [ 2041.644155] LustreError: 130667:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325460 with bad export cookie 16844581468692726292 [ 2041.646735] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2041.650665] LustreError: 130667:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2041.799450] Lustre: server umount lustre-MDT0001 complete [ 2045.278344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2047.265480] Lustre: DEBUG MARKER: files compared the same [ 2058.471138] Lustre: server umount lustre-OST0000 complete [ 2067.182705] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2068.420809] Key type lgssc unregistered [ 2068.584105] LNet: 141551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.589595] LNetError: 141551:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.597616] LNet: Removed LNI 192.168.204.123@tcp [ 2069.002563] Key type .llcrypt unregistered [ 2069.003768] Key type ._llcrypt unregistered [ 2106.303356] Key type ._llcrypt registered [ 2106.304888] Key type .llcrypt registered [ 2106.352453] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2106.771988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2106.825206] alg: No test for adler32 (adler32-zlib) [ 2107.709925] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 2107.823405] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2109.424171] Key type lgssc registered [ 2109.920679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2113.195197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2113.212237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2114.439483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2116.029141] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug +malloc all [ 2120.955951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2121.075972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2122.746205] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug +malloc all [ 2124.942386] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2126.276666] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2130.658166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2130.827334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2133.222281] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug +malloc all [ 2135.217196] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2139.047452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2144.228345] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2144.228428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.229055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2144.241542] Lustre: Skipped 1 previous similar message [ 2145.455062] LustreError: 148631:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2145.493491] Lustre: server umount lustre-OST0000 complete [ 2149.350026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.355566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.360322] Lustre: Skipped 1 previous similar message [ 2149.363099] Lustre: Skipped 1 previous similar message [ 2149.590874] LustreError: 148882:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2149.600657] LustreError: 148882:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2149.656951] Lustre: server umount lustre-MDT0000 complete [ 2151.862657] LustreError: 145983:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325571 with bad export cookie 16430522800511200715 [ 2151.864760] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2151.868857] LustreError: 145983:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2151.954918] LustreError: 149131:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2151.957946] LustreError: 149131:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2152.033063] Lustre: server umount lustre-MDT0001 complete [ 2157.504101] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2158.805899] Key type lgssc unregistered [ 2158.959752] LNet: 149806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.962601] LNetError: 149806:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2158.968341] LNet: Removed LNI 192.168.204.123@tcp [ 2159.318287] Key type .llcrypt unregistered [ 2159.320389] Key type ._llcrypt unregistered [ 2192.072229] Key type ._llcrypt registered [ 2192.075352] Key type .llcrypt registered [ 2192.129140] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2192.520563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2192.572297] alg: No test for adler32 (adler32-zlib) [ 2193.462922] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 2193.554643] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2195.144182] Key type lgssc registered [ 2195.618483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2198.709392] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2198.721675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2214.176915] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2219.296275] LustreError: 154227:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.123@tcp: failed processing log, type 4: rc = -110 [ 2245.920150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2248.490515] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2251.671057] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2256.288069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.329930] Lustre: *** cfs_fail_loc=706, val=0*** [ 2256.517956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2256.550378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2258.042507] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2263.172846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2263.306870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2264.729756] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2266.750689] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2268.175285] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2272.224880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2272.228124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.233271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2273.761928] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2273.762734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2273.766573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2274.989110] LustreError: 156936:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2275.027694] Lustre: server umount lustre-OST0000 complete [ 2278.881538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2278.883228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.884799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2278.884805] Lustre: Skipped 1 previous similar message [ 2278.897048] Lustre: Skipped 1 previous similar message [ 2284.001487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2284.003667] Lustre: Skipped 2 previous similar messages [ 2291.168140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2291.203429] LustreError: 157185:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2291.205795] LustreError: 157185:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2291.241621] Lustre: server umount lustre-MDT0000 complete [ 2292.193914] LustreError: 155395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.203171] LustreError: 155395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2293.139417] LustreError: 154236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325712 with bad export cookie 6919495558090990429 [ 2293.142153] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2293.147608] LustreError: 154236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2293.221337] LustreError: 157436:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2293.223955] LustreError: 157436:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2293.300156] Lustre: server umount lustre-MDT0001 complete [ 2298.460531] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2299.561243] Key type lgssc unregistered [ 2299.698443] LNet: 158112:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2299.701384] LNetError: 158112:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2299.711419] LNet: Removed LNI 192.168.204.123@tcp [ 2300.032751] Key type .llcrypt unregistered [ 2300.034221] Key type ._llcrypt unregistered [ 2334.601641] Key type ._llcrypt registered [ 2334.603228] Key type .llcrypt registered [ 2334.645934] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2335.131457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2335.140323] alg: No test for adler32 (adler32-zlib) [ 2336.022117] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 2336.117632] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2337.712197] Key type lgssc registered [ 2338.196528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2341.827849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2341.841561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.441080] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2348.643673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2348.844601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2350.301236] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2354.145361] LustreError: 163615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.150134] LustreError: 163615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2354.650561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2354.764701] LustreError: 163615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.771546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2356.755416] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2358.881029] LustreError: 163620:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2358.887840] LustreError: 163620:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2361.052859] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2361.058086] Lustre: MGS: Client f630b7c8-302e-423e-990d-d7aa67862c20 (at 0@lo) reconnecting [ 2361.147394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2361.180656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2362.606117] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2371.554702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.555786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2371.562621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2371.568568] Lustre: Skipped 1 previous similar message [ 2376.680486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2376.684722] Lustre: Skipped 1 previous similar message [ 2377.010740] LustreError: 165965:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2377.061561] Lustre: server umount lustre-OST0000 complete [ 2381.794977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2381.795023] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2381.802205] Lustre: Skipped 1 previous similar message [ 2381.804592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2381.806728] Lustre: Skipped 1 previous similar message [ 2385.214289] LustreError: 166216:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2385.217013] LustreError: 166216:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2385.231309] Lustre: server umount lustre-MDT0000 complete [ 2386.956509] LustreError: 166467:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2386.959319] LustreError: 166467:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2387.017139] Lustre: server umount lustre-MDT0001 complete [ 2388.797688] Lustre: server umount MGS complete [ 2393.353135] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 22:50:13 (1767325813) [ 2416.849401] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 2440.939932] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2448.292713] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2448.660149] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2449.450110] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2450.432889] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2455.735072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2455.757595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2457.151927] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2464.156926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2464.183221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2464.285189] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2464.314248] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2464.354653] Lustre: lustre-MDT0001: new disk, initializing [ 2464.384040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2464.396229] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2464.399276] Lustre: 177708:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2464.402272] LustreError: 177708:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2465.981997] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2473.570873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2473.618981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2473.728620] Lustre: lustre-OST0000: new disk, initializing [ 2473.731159] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2473.759901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2474.913860] Lustre: 178831:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000b125f8f4 x1853175370909056/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1767325910 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2475.917353] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2479.347035] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2483.336854] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2483.343019] Lustre: MGS: Client c898cfb1-5395-449f-b31f-269cc1e23a77 (at 0@lo) reconnecting [ 2483.364360] Lustre: 176649:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2483.373721] LustreError: 176649:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2483.384500] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2483.424480] Lustre: lustre-MDT0000: new disk, initializing [ 2483.473479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2483.482686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2484.000645] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2484.512951] Lustre: 179960:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000c498fc73 x1853175370914944/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1767325919 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2484.979907] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2493.603421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2493.606956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2493.674262] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2514.401215] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2514.403909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2514.408325] Lustre: Skipped 1 previous similar message [ 2514.410679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2520.431115] LustreError: 180479:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2520.433539] LustreError: 180479:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2520.479602] Lustre: server umount lustre-OST0000 complete [ 2526.689938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2526.690530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2526.693805] Lustre: Skipped 2 previous similar messages [ 2526.696179] Lustre: Skipped 3 previous similar messages [ 2526.698736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2526.701330] LustreError: Skipped 1 previous similar message [ 2528.449842] LustreError: 180730:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2528.452310] LustreError: 180730:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2528.461900] Lustre: server umount lustre-MDT0000 complete [ 2536.298446] Lustre: server umount lustre-MDT0001 complete [ 2538.031262] LustreError: 181229:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2538.033820] LustreError: 181229:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2538.055225] Lustre: server umount MGS complete [ 2542.621613] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 22:52:42 (1767325962) [ 2550.784417] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2551.939700] Key type lgssc unregistered [ 2552.084509] LNet: 182852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2552.087319] LNetError: 182852:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2552.100516] LNet: Removed LNI 192.168.204.123@tcp [ 2552.431066] Key type .llcrypt unregistered [ 2552.432298] Key type ._llcrypt unregistered [ 2581.527909] Key type ._llcrypt registered [ 2581.529305] Key type .llcrypt registered [ 2581.575981] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2582.098571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2582.107725] alg: No test for adler32 (adler32-zlib) [ 2582.968911] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 2583.060121] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2584.648266] Key type lgssc registered [ 2585.068589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2588.014386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2588.021407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.189258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.645716] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2595.301179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2595.434932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2597.011766] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2599.112338] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2600.549335] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2604.508948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2604.668353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2606.699711] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2608.472854] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2613.862931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2619.361491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2619.362861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2619.364040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2619.371256] Lustre: Skipped 1 previous similar message [ 2622.382106] LustreError: 189735:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2622.419277] Lustre: server umount lustre-OST0000 complete [ 2624.481226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2624.481824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2624.485259] Lustre: Skipped 1 previous similar message [ 2624.487524] Lustre: Skipped 1 previous similar message [ 2629.601103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2629.603479] Lustre: Skipped 2 previous similar messages [ 2630.207702] LustreError: 189984:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2630.209807] LustreError: 189984:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2630.242181] Lustre: server umount lustre-MDT0000 complete [ 2632.091332] LustreError: 187086:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326051 with bad export cookie 8040275589867469749 [ 2632.095158] LustreError: 187086:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2632.096988] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2632.166714] LustreError: 190236:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2632.171209] LustreError: 190236:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2632.252749] Lustre: server umount lustre-MDT0001 complete [ 2637.322826] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2638.463603] Key type lgssc unregistered [ 2638.621406] LNet: 190911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2638.625441] LNetError: 190911:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2638.635369] LNet: Removed LNI 192.168.204.123@tcp [ 2638.954672] Key type .llcrypt unregistered [ 2638.956080] Key type ._llcrypt unregistered [ 2664.974796] Key type ._llcrypt registered [ 2664.975876] Key type .llcrypt registered [ 2665.012743] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2665.418775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2665.440047] alg: No test for adler32 (adler32-zlib) [ 2666.290930] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 2666.374223] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2667.952152] Key type lgssc registered [ 2668.340982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2670.850294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2670.853494] LustreError: 194881:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2670.856091] LustreError: 194882:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2670.859976] LustreError: 194881:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 2670.862286] LustreError: 194881:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2670.865939] LustreError: 194881:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2670.866527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.022823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2675.694997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2675.790743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2677.172045] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2678.756727] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing lsmod [ 2680.042318] LustreError: 196298:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2680.042690] LustreError: 196297:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2680.046747] LustreError: 196298:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4998 [ 2680.048946] LustreError: 196298:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2680.053102] LustreError: 196298:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2680.053168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2680.181245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2685.409888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2685.411156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2685.413800] Lustre: Skipped 2 previous similar messages [ 2685.415799] Lustre: Skipped 1 previous similar message [ 2686.434540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2686.437226] Lustre: Skipped 3 previous similar messages [ 2687.547733] LustreError: 196536:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2687.573122] Lustre: server umount lustre-MDT0000 complete [ 2689.124297] LustreError: 194898:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326108 with bad export cookie 7454406805414198414 [ 2689.127696] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2689.128297] LustreError: 194898:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2690.528172] LustreError: 195545:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009e97cb92 x1853175716920704/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2690.540724] LustreError: 196786:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2690.543361] LustreError: 196786:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2690.592662] Lustre: server umount lustre-MDT0001 complete [ 2702.130901] LustreError: 196988:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2702.133427] LustreError: 196988:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2702.175686] Lustre: server umount lustre-OST0000 complete [ 2706.282152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2706.440976] LustreError: 197657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.447455] LustreError: 197657:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2706.465330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2707.805060] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2711.521480] LustreError: 197657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.860976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2711.992270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2713.337133] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2715.216885] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2716.438283] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2719.930786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2720.041353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2722.055930] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2723.901060] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2731.302952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2736.611332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2736.611755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2736.615913] Lustre: Skipped 1 previous similar message [ 2736.616139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2737.646410] LustreError: 200307:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2737.649622] LustreError: 200307:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2737.686585] Lustre: server umount lustre-OST0000 complete [ 2741.466311] Lustre: server umount lustre-MDT0000 complete [ 2741.730452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.731256] LustreError: 199057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2741.735141] Lustre: Skipped 1 previous similar message [ 2741.747044] LustreError: 199057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2743.009098] LustreError: 197681:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326162 with bad export cookie 7454406805414199086 [ 2743.011967] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2743.013245] LustreError: 197681:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2743.084526] LustreError: 200806:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2743.086444] LustreError: 200806:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2743.133140] Lustre: server umount lustre-MDT0001 complete [ 2747.756551] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2748.846530] Key type lgssc unregistered [ 2748.981772] LNet: 201481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2748.985953] LNetError: 201481:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2748.996273] LNet: Removed LNI 192.168.204.123@tcp [ 2749.328257] Key type .llcrypt unregistered [ 2749.329453] Key type ._llcrypt unregistered [ 2777.670864] Key type ._llcrypt registered [ 2777.671749] Key type .llcrypt registered [ 2777.706131] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2778.006157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2778.035683] alg: No test for adler32 (adler32-zlib) [ 2778.886387] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 2778.967530] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2780.544176] Key type lgssc registered [ 2780.912710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2783.599355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2783.607747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2784.777194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.177515] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2790.371459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2790.509041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2791.915910] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2793.685309] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2794.857824] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2798.370121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2798.474712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2800.420723] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2802.247787] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2809.767679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2814.562951] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2815.329773] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2820.068073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2820.068401] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2820.069856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2820.073847] Lustre: Skipped 1 previous similar message [ 2822.381152] LustreError: 208685:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2822.415222] Lustre: server umount lustre-OST0000 complete [ 2825.187208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.188576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2825.194524] Lustre: Skipped 1 previous similar message [ 2825.197988] Lustre: Skipped 2 previous similar messages [ 2830.076149] LustreError: 208939:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2830.078623] LustreError: 208939:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2830.101810] Lustre: server umount lustre-MDT0000 complete [ 2830.304945] LustreError: 205932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.310992] LustreError: 205932:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2831.594133] LustreError: 205912:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326250 with bad export cookie 15466721723803396170 [ 2831.597268] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2831.599475] LustreError: 205912:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2831.663657] LustreError: 209188:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2831.665479] LustreError: 209188:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2831.715510] Lustre: server umount lustre-MDT0001 complete [ 2836.058986] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2836.987787] Key type lgssc unregistered [ 2837.114386] LNet: 209863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2837.116571] LNetError: 209863:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2837.128275] LNet: Removed LNI 192.168.204.123@tcp [ 2837.391787] Key type .llcrypt unregistered [ 2837.392875] Key type ._llcrypt unregistered [ 2862.249707] Key type ._llcrypt registered [ 2862.251070] Key type .llcrypt registered [ 2862.286413] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2862.580311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2862.621064] alg: No test for adler32 (adler32-zlib) [ 2863.464353] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 2863.539827] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2865.120123] Key type lgssc registered [ 2865.471234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2867.981751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2867.987080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2869.132397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2870.424240] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2874.537873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2874.675582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2876.033170] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2877.783232] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2878.988612] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2882.627200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2882.749167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2884.696657] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2886.517665] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2894.055538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2904.545250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2904.546075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2904.548012] LustreError: Skipped 1 previous similar message [ 2904.551747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2904.555213] Lustre: Skipped 1 previous similar message [ 2905.709653] LustreError: 216767:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2905.741537] Lustre: server umount lustre-OST0000 complete [ 2909.666484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.667024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2909.670578] Lustre: Skipped 2 previous similar messages [ 2913.404261] LustreError: 217016:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2913.406330] LustreError: 217016:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2913.434217] Lustre: server umount lustre-MDT0000 complete [ 2914.784857] LustreError: 214116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2914.789906] LustreError: 214116:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2914.904368] LustreError: 214095:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326334 with bad export cookie 4762512926838548658 [ 2914.905995] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2914.909948] LustreError: 214095:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2914.967858] LustreError: 217266:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2914.969832] LustreError: 217266:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2915.018034] Lustre: server umount lustre-MDT0001 complete [ 2919.128331] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2920.036744] Key type lgssc unregistered [ 2920.161601] LNet: 217943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2920.164412] LNetError: 217943:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2920.176353] LNet: Removed LNI 192.168.204.123@tcp [ 2920.469086] Key type .llcrypt unregistered [ 2920.470434] Key type ._llcrypt unregistered [ 2946.852169] Key type ._llcrypt registered [ 2946.853288] Key type .llcrypt registered [ 2946.888453] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2947.267579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2947.285504] alg: No test for adler32 (adler32-zlib) [ 2948.127077] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 2948.197916] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2949.776185] Key type lgssc registered [ 2950.122238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2952.640465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2952.646446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2953.794429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2955.006875] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2958.831074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2958.949938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2960.195731] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2961.854655] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2962.961752] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2966.171063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2966.280411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2968.016273] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2969.629468] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2979.624657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2982.365693] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2983.530274] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2983.533097] Lustre: Skipped 2 previous similar messages [ 2984.829517] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2984.831938] Lustre: Skipped 2 previous similar messages [ 2990.095393] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2990.108431] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2996.378297] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.23@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2996.381612] Lustre: Skipped 2 previous similar messages [ 2998.519669] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.23@tcp 192.168.0.[2,10]@tcp [ 3003.360567] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.23/24@tcp in log lustre-MDT0000 [ 3003.362989] Lustre: Skipped 2 previous similar messages [ 3006.580404] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.23/24@tcp [ 3006.582392] Lustre: Skipped 1 previous similar message [ 3010.531402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.532133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3010.532414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3010.536692] Lustre: Skipped 1 previous similar message [ 3014.574704] LustreError: 226493:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3014.605484] Lustre: server umount lustre-OST0000 complete [ 3020.257089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3020.259290] LustreError: Skipped 1 previous similar message [ 3020.260310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3020.264093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3020.266646] Lustre: Skipped 1 previous similar message [ 3022.143629] LustreError: 226742:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3022.146207] LustreError: 226742:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3022.169868] Lustre: server umount lustre-MDT0000 complete [ 3023.769669] LustreError: 222374:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326443 with bad export cookie 7587573334442103552 [ 3023.772439] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3023.773629] LustreError: 222374:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3023.846729] LustreError: 226992:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3023.849201] LustreError: 226992:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3023.917808] Lustre: server umount lustre-MDT0001 complete [ 3028.320672] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 3029.316391] Key type lgssc unregistered [ 3029.454394] LNet: 227666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3029.456126] LNetError: 227666:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3029.463360] LNet: Removed LNI 192.168.204.123@tcp [ 3029.750829] Key type .llcrypt unregistered [ 3029.751965] Key type ._llcrypt unregistered [ 3058.416753] Key type ._llcrypt registered [ 3058.418361] Key type .llcrypt registered [ 3058.456539] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3058.789470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3058.845048] alg: No test for adler32 (adler32-zlib) [ 3059.695858] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 3059.773403] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 3061.352127] Key type lgssc registered [ 3061.688700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3064.120539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3064.126465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3065.260546] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3065.264478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3065.268393] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.23/24@tcp [ 3066.418112] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3070.175080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3070.276405] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3070.284677] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.23/24@tcp [ 3070.298196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3071.508558] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3073.268703] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3074.488136] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3077.699838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3077.810696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3079.576742] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3081.257356] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3090.088639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3095.521509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3095.522490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3095.524759] LustreError: Skipped 1 previous similar message [ 3095.530366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3095.533494] Lustre: Skipped 1 previous similar message [ 3096.494910] LustreError: 235026:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3096.527456] Lustre: server umount lustre-OST0000 complete [ 3100.641740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.642158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3100.645319] Lustre: Skipped 2 previous similar messages [ 3104.061718] LustreError: 235276:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3104.064237] LustreError: 235276:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3104.085885] Lustre: server umount lustre-MDT0000 complete [ 3105.544333] LustreError: 232301:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326524 with bad export cookie 6806724760823523123 [ 3105.547819] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3105.549161] LustreError: 232301:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3105.613659] LustreError: 235527:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3105.616449] LustreError: 235527:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3105.661174] Lustre: server umount lustre-MDT0001 complete [ 3109.695265] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 3110.653866] Key type lgssc unregistered [ 3110.783327] LNet: 236202:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3110.786027] LNetError: 236202:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3110.800330] LNet: Removed LNI 192.168.204.123@tcp [ 3111.072177] Key type .llcrypt unregistered [ 3111.072973] Key type ._llcrypt unregistered [ 3125.127454] Key type ._llcrypt registered [ 3125.129029] Key type .llcrypt registered [ 3125.167855] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 3147.578843] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3147.907943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3147.951520] alg: No test for adler32 (adler32-zlib) [ 3148.796713] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 3148.878719] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 3150.456096] Key type lgssc registered [ 3150.782285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3156.254507] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3156.533574] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3157.077996] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3157.852071] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3162.892286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3162.911350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3162.915473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.990924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3164.001911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3164.038438] Lustre: lustre-MDT0000: new disk, initializing [ 3164.062371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3164.068427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3165.312364] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3171.640273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3171.663638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3171.699325] Lustre: 245249:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3171.722589] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3171.724307] Lustre: Skipped 1 previous similar message [ 3171.763313] Lustre: lustre-MDT0001: new disk, initializing [ 3171.794640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3171.809417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3171.812858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3173.097649] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3176.149324] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3177.327112] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3181.427428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3181.451080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3181.540172] Lustre: lustre-OST0000: new disk, initializing [ 3181.542787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3181.568545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3183.271712] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3185.655573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3185.659911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3185.689436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3186.203318] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3187.457963] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3187.524945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3188.708252] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3188.769521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3190.753831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3190.755970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.757409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3190.761045] Lustre: Skipped 1 previous similar message [ 3191.407089] LustreError: 248875:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3191.445639] Lustre: server umount lustre-OST0000 complete [ 3195.873484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3195.874108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3195.877672] Lustre: Skipped 1 previous similar message [ 3195.878946] Lustre: Skipped 1 previous similar message [ 3200.992877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3200.994320] Lustre: Skipped 2 previous similar messages [ 3206.112922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3206.114897] Lustre: Skipped 1 previous similar message [ 3207.136111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3207.169704] LustreError: 249123:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3207.172256] LustreError: 249123:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3207.201970] Lustre: server umount lustre-MDT0000 complete [ 3208.614197] LustreError: 245240:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326627 with bad export cookie 4456000484401668824 [ 3208.617062] LustreError: 245240:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3208.617204] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3208.675980] LustreError: 249373:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3208.677522] LustreError: 249373:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3208.732095] Lustre: server umount lustre-MDT0001 complete [ 3212.494166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3212.641110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.763018] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3217.332689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3217.435179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3218.586948] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3220.121162] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3221.136905] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3224.209074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3226.007852] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3227.589684] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3228.853023] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3239.091172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3240.264755] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3240.335119] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3245.693977] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3245.717823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3245.768772] Lustre: lustre-OST0001: new disk, initializing [ 3245.770725] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3245.798380] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3245.801872] Lustre: Skipped 1 previous similar message [ 3247.648249] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3250.858192] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3252.738399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3252.741950] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3252.771996] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3252.967290] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3254.230959] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3254.308311] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3255.426436] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93c80448d800.ost_server_uuid 50 [ 3255.911066] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff93c80448d800.ost_server_uuid in FULL state after 0 sec [ 3257.827257] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3257.827639] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3257.829994] LustreError: Skipped 2 previous similar messages [ 3257.833737] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3257.836290] Lustre: Skipped 1 previous similar message [ 3263.663078] LustreError: 255106:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3263.665063] LustreError: 255106:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3263.681961] Lustre: server umount lustre-OST0001 complete [ 3268.067269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3268.068103] LustreError: 252024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3268.068877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3268.068884] Lustre: Skipped 2 previous similar messages [ 3268.072155] Lustre: Skipped 2 previous similar messages [ 3268.078337] LustreError: 252024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3273.187241] LustreError: 255019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3273.191959] LustreError: 255019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3278.306553] LustreError: 252024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3278.312098] LustreError: 252024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3279.328128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3279.346584] LustreError: 255307:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3279.348107] LustreError: 255307:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3279.380093] Lustre: server umount lustre-OST0000 complete [ 3283.425819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.544880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3288.546256] Lustre: Skipped 8 previous similar messages [ 3295.200095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3295.229935] LustreError: 255558:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3295.232071] LustreError: 255558:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3295.258442] Lustre: server umount lustre-MDT0000 complete [ 3296.662210] LustreError: 253845:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326715 with bad export cookie 4456000484401669706 [ 3296.664857] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3296.665772] LustreError: 253845:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3296.772594] Lustre: server umount lustre-MDT0001 complete [ 3304.052478] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 3304.951870] Key type lgssc unregistered [ 3305.069317] LNet: 257038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3305.071821] LNetError: 257038:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3305.082274] LNet: Removed LNI 192.168.204.123@tcp [ 3305.351670] Key type .llcrypt unregistered [ 3305.352542] Key type ._llcrypt unregistered [ 3323.550732] Key type ._llcrypt registered [ 3323.551792] Key type .llcrypt registered [ 3323.583974] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 3345.478349] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3345.709327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3345.730979] alg: No test for adler32 (adler32-zlib) [ 3346.576815] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 3346.647559] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 3348.224101] Key type lgssc registered [ 3348.517844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3354.220563] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3354.412981] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3354.964577] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3355.698233] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3360.650732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3360.667656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3360.672123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.751087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3361.763559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3361.800387] Lustre: lustre-MDT0000: new disk, initializing [ 3361.824188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3361.831229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3363.070778] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3369.024387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3369.047499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3369.076698] Lustre: 267054:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3369.095942] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3369.097603] Lustre: Skipped 1 previous similar message [ 3369.128636] Lustre: lustre-MDT0001: new disk, initializing [ 3369.159145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3369.170399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3369.174212] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3370.335760] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3373.187620] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3374.211272] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3378.109203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3378.132985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3378.217358] Lustre: lustre-OST0000: new disk, initializing [ 3378.219490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3378.245638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3379.990945] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3382.855454] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3383.796623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3383.798901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3383.828942] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3384.024964] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3384.090202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3385.234864] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3385.298781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3388.898183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3388.899558] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3388.900419] LustreError: Skipped 1 previous similar message [ 3388.904163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3388.906629] Lustre: Skipped 1 previous similar message [ 3391.987567] LustreError: 270683:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3392.023264] Lustre: server umount lustre-OST0000 complete [ 3394.017760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3394.018270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3394.020724] Lustre: Skipped 2 previous similar messages [ 3399.137243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3399.140069] Lustre: Skipped 2 previous similar messages [ 3399.551324] LustreError: 270934:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3399.553317] LustreError: 270934:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3399.579139] Lustre: server umount lustre-MDT0000 complete [ 3401.105494] LustreError: 267090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326820 with bad export cookie 12350204287186758325 [ 3401.108505] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3401.109433] LustreError: 267090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3401.171524] LustreError: 271185:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3401.173207] LustreError: 271185:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3401.219462] Lustre: server umount lustre-MDT0001 complete [ 3403.921994] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 23:07:03 (1767326823) [ 3414.641980] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 3433.985542] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3439.660804] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3440.069537] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3440.650151] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3441.626560] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3446.663876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3446.685758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3446.761930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3446.774194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3446.811352] Lustre: lustre-MDT0000: new disk, initializing [ 3446.833687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3446.840150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3448.078663] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3454.192293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3454.215487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3454.247645] Lustre: 279870:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3454.268925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3454.270586] Lustre: Skipped 1 previous similar message [ 3454.302210] Lustre: lustre-MDT0001: new disk, initializing [ 3454.330970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3454.341087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3454.344117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3455.598772] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3458.590197] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3459.694419] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3463.982307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3464.004511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3464.085590] Lustre: lustre-OST0000: new disk, initializing [ 3464.087907] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3464.112772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3465.893129] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3468.884279] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3470.837636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3470.840552] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3470.865930] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3475.891204] Lustre: Failing over lustre-OST0000 [ 3475.909087] LustreError: 283001:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3475.910934] LustreError: 283001:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3475.947606] Lustre: server umount lustre-OST0000 complete [ 3491.176314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3491.278182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3491.282048] Lustre: Skipped 1 previous similar message [ 3491.287353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3491.293111] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3492.320095] Lustre: 263530:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767326895/real 1767326895] req@0000000018ef263f x1853176429994240/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1767326911 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3492.329157] Lustre: 263530:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 3493.118130] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3494.319376] Lustre: Failing over lustre-MDT0000 [ 3494.378896] LustreError: 284405:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3494.381366] LustreError: 284405:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3494.410328] Lustre: server umount lustre-MDT0000 complete [ 3495.393818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3495.396033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3495.632396] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3497.440155] Lustre: 263531:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767326900/real 1767326900] req@000000001ef95649 x1853176429996160/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1767326916 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3500.745121] LustreError: 282141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3500.749871] LustreError: 282141:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3501.537095] LustreError: 279881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3501.543518] LustreError: 279881:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3502.560126] Lustre: 263531:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767326905/real 1767326905] req@0000000099159631 x1853176429997312/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1767326921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3505.865344] LustreError: 279877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3510.262654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3510.299150] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3510.303176] Lustre: 279873:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.204.123@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3510.364286] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3510.400969] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3510.985538] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3511.615300] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3515.879657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3515.886531] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3515.888933] Lustre: Skipped 1 previous similar message [ 3515.903925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3520.994547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3520.996360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3520.997178] LustreError: Skipped 1 previous similar message [ 3521.001645] Lustre: Skipped 2 previous similar messages [ 3521.003340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3521.005945] Lustre: Skipped 1 previous similar message [ 3522.924167] LustreError: 285845:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3522.925926] LustreError: 285845:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3522.957716] Lustre: server umount lustre-OST0000 complete [ 3526.113570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3526.114162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3526.117406] Lustre: Skipped 1 previous similar message [ 3531.413994] Lustre: server umount lustre-MDT0000 complete [ 3532.796042] LustreError: 282306:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326952 with bad export cookie 12350204287186760376 [ 3532.798451] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3532.799305] LustreError: 282306:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3532.907934] Lustre: server umount lustre-MDT0001 complete [ 3537.019125] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 3538.035571] Key type lgssc unregistered [ 3538.152294] LNet: 287019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3538.154144] LNetError: 287019:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3538.161247] LNet: Removed LNI 192.168.204.123@tcp [ 3538.430945] Key type .llcrypt unregistered [ 3538.432423] Key type ._llcrypt unregistered [ 3578.034590] Key type ._llcrypt registered [ 3578.035417] Key type .llcrypt registered [ 3578.074258] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3578.508618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3578.549610] alg: No test for adler32 (adler32-zlib) [ 3579.392071] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 3579.462676] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 3581.040131] Key type lgssc registered [ 3581.373234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3584.843074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3584.849884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3585.986774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3587.180051] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3590.074644] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3591.972886] LustreError: 292386:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3591.977546] LustreError: 292386:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3591.980285] LustreError: 292386:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 3592.042381] LustreError: 292908:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3592.093716] Lustre: server umount lustre-MDT0000 complete [ 3602.223308] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 3622.914758] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3628.580290] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3628.739908] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3629.293478] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3629.978360] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3634.912149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3634.936849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3635.037302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3635.047751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3635.079462] Lustre: lustre-MDT0000: new disk, initializing [ 3635.104865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3635.111972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3636.344401] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3642.242056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3642.261629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3642.296670] Lustre: 301251:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3642.318119] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3642.320426] Lustre: Skipped 1 previous similar message [ 3642.355190] Lustre: lustre-MDT0001: new disk, initializing [ 3642.384927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3642.395324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3642.398274] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3643.627687] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3646.690640] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3647.991459] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3652.098704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3652.122834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3652.219056] Lustre: lustre-OST0000: new disk, initializing [ 3652.220694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3652.243518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3654.083309] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3657.105032] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3660.277586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3660.281040] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3660.304544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3664.213595] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3664.740037] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 23:11:24 (1767327084) [ 3729.376470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3729.378480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3729.382134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3731.936391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3731.936632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3731.939016] Lustre: Skipped 1 previous similar message [ 3731.942701] Lustre: Skipped 1 previous similar message [ 3733.691254] LustreError: 306018:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3733.692790] LustreError: 306018:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3733.727717] Lustre: server umount lustre-MDT0000 complete [ 3736.217415] LustreError: 301241:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327155 with bad export cookie 5953241008311043569 [ 3736.218474] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3736.221228] LustreError: 301241:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3737.056347] LustreError: 305913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3737.056374] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3737.061461] LustreError: 305913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3737.065032] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3741.088329] LustreError: 301257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3741.088382] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3742.329147] LustreError: 306470:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3742.331202] LustreError: 306470:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3742.367279] Lustre: server umount lustre-MDT0001 complete [ 3760.096079] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3760.109625] LustreError: 307064:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3760.111066] LustreError: 307064:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3760.143670] Lustre: server umount lustre-OST0000 complete [ 3765.372305] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 3784.203157] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3789.472883] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3789.802039] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3790.415938] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3791.078409] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3796.105175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3796.129501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3796.219070] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3796.229524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3796.267634] Lustre: lustre-MDT0000: new disk, initializing [ 3796.290832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3796.297066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3797.660259] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3803.969209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3803.990754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3804.021800] Lustre: 314460:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3804.044706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3804.048034] Lustre: Skipped 1 previous similar message [ 3804.086212] Lustre: lustre-MDT0001: new disk, initializing [ 3804.126946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3804.131388] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3805.370386] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3808.452621] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3809.641181] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3813.994811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3814.015915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3814.106951] Lustre: lustre-OST0000: new disk, initializing [ 3814.109082] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3814.136051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.138939] Lustre: Skipped 1 previous similar message [ 3816.045209] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3817.465365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3817.468601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3817.499045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3819.126757] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3820.499656] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3820.575663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3821.910827] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3821.985741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3827.683949] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3827.684173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3827.684689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3827.686976] LustreError: Skipped 1 previous similar message [ 3827.690710] Lustre: Skipped 1 previous similar message [ 3828.718102] LustreError: 318090:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3828.720460] LustreError: 318090:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3828.759983] Lustre: server umount lustre-OST0000 complete [ 3832.801686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.805227] Lustre: Skipped 1 previous similar message [ 3836.497406] Lustre: server umount lustre-MDT0000 complete [ 3837.920897] LustreError: 314470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3837.926503] LustreError: 314470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3837.936161] LustreError: 315586:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327257 with bad export cookie 5953241008311202385 [ 3837.938964] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3837.939789] LustreError: 315586:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3838.005487] LustreError: 318589:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3838.007324] LustreError: 318589:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3838.057296] Lustre: server umount lustre-MDT0001 complete [ 3840.552457] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:14:20 (1767327260) [ 3851.423664] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 3870.391442] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3876.256916] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3876.543636] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3877.350528] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3878.142530] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3883.142975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3883.168672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3883.244176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3883.255326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3883.290117] Lustre: lustre-MDT0000: new disk, initializing [ 3883.313762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3883.320362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3884.579748] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3890.595884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3890.618511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3890.646598] Lustre: 327273:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3890.731302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3891.904651] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3894.860389] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3895.951186] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3900.018891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3900.042230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3901.996082] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3903.617472] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3903.656559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3904.924631] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3909.090396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3909.092048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3909.092820] LustreError: Skipped 1 previous similar message [ 3909.096374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3909.099580] Lustre: Skipped 4 previous similar messages [ 3912.686025] LustreError: 330512:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3912.687908] LustreError: 330512:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3912.722478] Lustre: server umount lustre-OST0000 complete [ 3920.291535] Lustre: server umount lustre-MDT0000 complete [ 3921.726309] LustreError: 328403:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327341 with bad export cookie 5953241008311203267 [ 3921.728495] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3921.729209] LustreError: 328403:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3921.839368] Lustre: server umount lustre-MDT0001 complete [ 3924.333103] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:15:44 (1767327344) [ 3934.857582] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 3953.396185] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3959.349272] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3959.540228] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3960.061191] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3960.749404] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3965.466372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3965.491052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3965.576499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3965.585947] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3965.587437] Lustre: Skipped 3 previous similar messages [ 3965.623037] Lustre: lustre-MDT0000: new disk, initializing [ 3965.624103] Lustre: Skipped 2 previous similar messages [ 3965.645904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3965.647489] Lustre: Skipped 2 previous similar messages [ 3965.652706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3965.655222] Lustre: Skipped 2 previous similar messages [ 3966.834713] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3972.598292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3972.618252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3972.649778] Lustre: 339705:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3972.740072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3973.895145] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3976.807747] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3977.842074] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3981.676868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3981.699579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3983.381926] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3983.440646] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3986.281878] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3988.449457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3988.450528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3988.451593] LustreError: Skipped 1 previous similar message [ 3988.455277] Lustre: Skipped 3 previous similar messages [ 3988.456106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3988.460551] Lustre: Skipped 5 previous similar messages [ 3993.966292] LustreError: 342941:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3993.967951] LustreError: 342941:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3994.002562] Lustre: server umount lustre-OST0000 complete [ 3998.689511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4001.501239] Lustre: server umount lustre-MDT0000 complete [ 4003.060046] LustreError: 343192:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327422 with bad export cookie 5953241008311204345 [ 4003.063526] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4003.063733] LustreError: 343192:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4003.182153] Lustre: server umount lustre-MDT0001 complete [ 4007.599399] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4008.673869] Key type lgssc unregistered [ 4008.798325] LNet: 344116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4008.801436] LNetError: 344116:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4008.810297] LNet: Removed LNI 192.168.204.123@tcp [ 4009.092575] Key type .llcrypt unregistered [ 4009.093704] Key type ._llcrypt unregistered [ 4036.135589] Key type ._llcrypt registered [ 4036.137912] Key type .llcrypt registered [ 4036.174450] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4036.568578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4036.591730] alg: No test for adler32 (adler32-zlib) [ 4037.446210] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 4037.528672] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4039.104140] Key type lgssc registered [ 4039.478577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4042.078069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4042.084583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4043.245632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4044.606736] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4048.606603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4048.729048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4050.023256] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4051.784400] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4052.945233] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4056.362777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4056.476522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4058.345243] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4060.176656] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4067.751453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4083.169778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4083.171904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4083.176650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4083.178700] Lustre: Skipped 1 previous similar message [ 4088.291095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4088.293369] Lustre: Skipped 1 previous similar message [ 4088.305098] LustreError: 351218:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4088.342759] Lustre: server umount lustre-OST0000 complete [ 4093.409691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4093.410504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4093.413387] Lustre: Skipped 2 previous similar messages [ 4098.529372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4098.532315] Lustre: Skipped 2 previous similar messages [ 4103.649173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4103.651649] Lustre: Skipped 1 previous similar message [ 4104.160119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4104.199677] LustreError: 351469:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4104.202324] LustreError: 351469:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4104.229159] Lustre: server umount lustre-MDT0000 complete [ 4105.784313] LustreError: 348589:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327525 with bad export cookie 12372474452908267265 [ 4105.786045] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4105.790651] LustreError: 348589:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4105.854491] LustreError: 351721:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4105.856334] LustreError: 351721:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4105.908227] Lustre: server umount lustre-MDT0001 complete [ 4110.449259] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4111.533547] Key type lgssc unregistered [ 4111.659364] LNet: 352395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4111.661567] LNetError: 352395:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4111.669370] LNet: Removed LNI 192.168.204.123@tcp [ 4111.976116] Key type .llcrypt unregistered [ 4111.977233] Key type ._llcrypt unregistered [ 4138.632103] Key type ._llcrypt registered [ 4138.632890] Key type .llcrypt registered [ 4138.664394] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4138.962136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4139.002126] alg: No test for adler32 (adler32-zlib) [ 4139.848965] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 4139.918880] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4141.496134] Key type lgssc registered [ 4141.826697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4144.252050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4144.258051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4145.402944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4146.602501] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4150.330814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4150.435326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4151.576666] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4153.145080] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4154.239639] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4157.456160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4157.558269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4159.249214] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4160.805848] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4169.831065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4175.329922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4175.330918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4175.332326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4175.335760] Lustre: Skipped 1 previous similar message [ 4175.342034] Lustre: Skipped 1 previous similar message [ 4176.592531] Lustre: lustre-OST0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 4179.630907] LustreError: 359494:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4179.663450] Lustre: server umount lustre-OST0000 complete [ 4181.261970] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff93c808a43800.ost_server_uuid 50 [ 4181.715900] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93c808a43800.ost_server_uuid in DISCONN state after 0 sec [ 4182.167975] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4192.445412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4192.446213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4192.448941] Lustre: Skipped 1 previous similar message [ 4195.553512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4195.555460] Lustre: Skipped 2 previous similar messages [ 4198.594322] LustreError: 359939:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4198.596939] LustreError: 359939:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4198.625695] Lustre: server umount lustre-MDT0000 complete [ 4200.182204] LustreError: 358846:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327619 with bad export cookie 1417544728811734097 [ 4200.184949] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4200.186435] LustreError: 358846:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4200.252212] LustreError: 360189:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4200.254314] LustreError: 360189:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4200.303965] Lustre: server umount lustre-MDT0001 complete [ 4203.017310] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 23:20:22 (1767327622) [ 4206.355759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4206.509626] LustreError: 361051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4206.517596] LustreError: 361051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4206.535745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4207.819697] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4211.681034] LustreError: 361051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4211.873895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4211.999288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4213.333644] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4215.180527] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4216.367987] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4219.772186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4221.585199] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4223.263591] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4227.342498] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4227.364979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4227.425271] Lustre: lustre-OST0001: new disk, initializing [ 4227.428167] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4227.457663] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4227.460178] Lustre: Skipped 1 previous similar message [ 4229.139882] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4231.592743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4232.128419] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4233.714712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4233.722412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4233.723864] Lustre: Skipped 1 previous similar message [ 4233.725768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4233.728666] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4233.766801] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4239.084031] LustreError: 364972:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4239.086659] LustreError: 364972:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4239.103432] Lustre: server umount lustre-OST0000 complete [ 4241.006820] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4241.080554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4242.357219] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4242.433644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4242.634331] LustreError: 363030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4247.754358] LustreError: 363029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4247.759673] LustreError: 363029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4254.179371] LustreError: 365597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4254.180479] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4254.180561] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4254.180569] Lustre: Skipped 1 previous similar message [ 4254.181425] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4254.181430] Lustre: Skipped 3 previous similar messages [ 4254.183680] LustreError: 365597:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4254.185494] LustreError: Skipped 1 previous similar message [ 4258.478225] LustreError: 365683:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4258.481403] LustreError: 365683:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4258.520519] Lustre: server umount lustre-OST0001 complete [ 4263.392980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4263.395974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4263.400352] Lustre: Skipped 1 previous similar message [ 4266.202878] Lustre: server umount lustre-MDT0000 complete [ 4267.751120] LustreError: 361031:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327687 with bad export cookie 1417544728811735154 [ 4267.754127] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4267.755914] LustreError: 361031:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4267.827296] LustreError: 366183:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4267.829266] LustreError: 366183:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 4267.883383] Lustre: server umount lustre-MDT0001 complete [ 4270.116468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4271.551517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4272.975238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4274.461749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4276.465538] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:21:36 (1767327696) [ 4279.570970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4279.655831] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4279.667498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4279.732517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4280.888231] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4284.543920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4284.570111] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4284.584048] Lustre: 368057:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4285.816288] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4287.361435] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4288.395091] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4291.414236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4291.469426] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4293.151020] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4294.711489] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4296.716289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4297.765863] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4297.786477] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4299.613395] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4301.405409] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4303.331802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4303.332798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4303.334878] Lustre: Skipped 2 previous similar messages [ 4303.337092] Lustre: Skipped 4 previous similar messages [ 4308.334859] LustreError: 371816:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4308.336763] LustreError: 371816:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4308.354274] Lustre: server umount lustre-OST0000 complete [ 4309.476263] LustreError: 370972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4309.482903] LustreError: 370972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4319.714685] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4319.716805] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4319.721225] Lustre: Skipped 1 previous similar message [ 4325.073804] Lustre: server umount lustre-OST0001 complete [ 4332.828309] Lustre: server umount lustre-MDT0000 complete [ 4334.317033] LustreError: 368981:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327753 with bad export cookie 1417544728811736512 [ 4334.320061] LustreError: 368981:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4334.320473] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4334.434428] Lustre: server umount lustre-MDT0001 complete [ 4336.531667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4338.124291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4339.723971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4341.247249] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4343.170784] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:22:42 (1767327762) [ 4355.520643] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 4374.437360] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4380.068333] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4380.277310] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4380.752017] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4381.468891] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4386.309022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4386.327951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4386.402847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4386.412437] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4386.443632] Lustre: lustre-MDT0000: new disk, initializing [ 4386.464139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4386.465974] Lustre: Skipped 3 previous similar messages [ 4386.470434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4387.671247] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4393.513637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4393.534074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4393.564438] Lustre: 382438:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4393.585509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4393.587109] Lustre: Skipped 1 previous similar message [ 4393.622694] Lustre: lustre-MDT0001: new disk, initializing [ 4393.664098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4393.668353] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4394.914608] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4397.718498] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4398.867759] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4403.285957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4403.307821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4403.399964] Lustre: lustre-OST0000: new disk, initializing [ 4403.401772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4405.229126] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4405.368035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4405.373091] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4405.403069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4408.351640] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4409.671718] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4409.744606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4411.042187] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4411.121647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4415.971689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4415.971793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4415.972770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4415.972774] Lustre: Skipped 10 previous similar messages [ 4415.975845] LustreError: Skipped 2 previous similar messages [ 4415.979920] Lustre: Skipped 3 previous similar messages [ 4426.208110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4426.230129] LustreError: 386070:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4426.232908] LustreError: 386070:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4426.271898] Lustre: server umount lustre-OST0000 complete [ 4429.792864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4442.080097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4442.141026] Lustre: server umount lustre-MDT0000 complete [ 4443.533089] LustreError: 383565:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327862 with bad export cookie 1417544728811738031 [ 4443.535551] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4443.536765] LustreError: 383565:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4443.638105] Lustre: server umount lustre-MDT0001 complete [ 4447.658599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4447.795612] LustreError: 387238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4447.800784] LustreError: 387238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4449.005020] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4452.706055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4452.811759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4452.814622] Lustre: Skipped 3 previous similar messages [ 4454.042608] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4455.653262] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4456.736681] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4459.868874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4461.661899] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4463.229150] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4464.577179] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4473.814460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 4475.078524] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4475.151311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4478.434106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4478.437437] LustreError: Skipped 1 previous similar message [ 4481.867909] Lustre: server umount lustre-OST0000 complete [ 4483.805876] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4485.177865] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4510.093162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4512.140155] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4513.992133] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4518.437290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4518.440717] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4518.440775] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4532.029357] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 17 sec [ 4533.729959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4533.730452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4533.732283] LustreError: Skipped 1 previous similar message [ 4533.735708] Lustre: Skipped 4 previous similar messages [ 4533.736188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4533.739710] Lustre: Skipped 15 previous similar messages [ 4539.054096] LustreError: 392279:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4539.056118] LustreError: 392279:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 4539.091543] Lustre: server umount lustre-OST0000 complete [ 4546.775136] Lustre: server umount lustre-MDT0000 complete [ 4548.321152] LustreError: 387218:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327967 with bad export cookie 1417544728811738913 [ 4548.324277] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4548.324958] LustreError: 387218:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4548.443325] Lustre: server umount lustre-MDT0001 complete [ 4551.265321] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:26:11 (1767327971) [ 4554.505292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4554.648286] LustreError: 393641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4555.905188] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4559.490948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4560.753137] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4562.429232] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4563.727468] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4567.087368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4569.058366] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4570.843533] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4572.274462] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4580.514151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 4581.903985] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4581.984768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4586.395318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4586.419963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4586.477523] Lustre: lustre-OST0001: new disk, initializing [ 4586.479689] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4586.508784] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4586.511844] Lustre: Skipped 5 previous similar messages [ 4588.410233] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4591.597982] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4592.932816] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4594.179605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4594.184626] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4594.212474] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4595.042145] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4596.471101] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4596.550529] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4599.269445] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4599.272457] LustreError: Skipped 1 previous similar message [ 4603.266625] Lustre: server umount lustre-OST0001 complete [ 4605.361834] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4605.444135] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4606.856501] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4606.929556] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4619.748317] LustreError: 395622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4619.757040] LustreError: 395622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 4631.773590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4633.698417] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4635.445418] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4635.955392] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4642.680382] Lustre: server umount lustre-OST0001 complete [ 4660.821449] Lustre: server umount lustre-OST0000 complete [ 4662.753898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4662.758599] Lustre: Skipped 7 previous similar messages [ 4662.760756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4662.762743] Lustre: Skipped 17 previous similar messages [ 4668.609151] LustreError: 400971:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4668.610904] LustreError: 400971:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 4668.642738] Lustre: server umount lustre-MDT0000 complete [ 4670.179121] LustreError: 393622:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767328089 with bad export cookie 1417544728811739865 [ 4670.182662] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4670.185758] LustreError: 393622:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4676.336769] Lustre: server umount lustre-MDT0001 complete [ 4678.456449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4680.018410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4681.618597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4683.198188] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4685.341596] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:28:25 (1767328105) [ 4688.784994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4688.882962] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4688.894847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4690.350329] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4694.369951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4694.403736] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4694.413751] Lustre: 403095:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4695.791368] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4697.522272] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4698.689749] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4702.056081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4702.127981] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4704.109815] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4705.878142] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4715.785149] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4715.807197] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4717.725328] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4719.455098] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4719.955559] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4721.238149] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4721.306070] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4722.592107] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4722.656608] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4723.857760] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93c80625c800.ost_server_uuid 50 [ 4724.350759] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff93c80625c800.ost_server_uuid in FULL state after 0 sec [ 4724.683045] Lustre: Permanently deactivating lustre-OST0001 [ 4724.688706] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4725.460085] Lustre: Permanently reactivating lustre-OST0001 [ 4727.778348] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4727.780230] LustreError: Skipped 3 previous similar messages [ 4732.478162] Lustre: server umount lustre-OST0001 complete [ 4739.790553] Lustre: server umount lustre-OST0000 complete [ 4747.414976] Lustre: server umount lustre-MDT0000 complete [ 4748.256901] LustreError: 403105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4748.262156] LustreError: 403105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 4748.888084] LustreError: 403086:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767328168 with bad export cookie 1417544728811741125 [ 4748.890054] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4748.893224] LustreError: 403086:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4749.009295] Lustre: server umount lustre-MDT0001 complete [ 4751.189231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4752.796457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4754.387074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4755.939336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4758.157741] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:29:37 (1767328177) [ 4758.774974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4762.107579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4762.198879] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4762.208395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4762.210953] Lustre: Skipped 5 previous similar messages [ 4763.629744] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4767.623738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4767.652184] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4767.661028] Lustre: 410347:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4769.062917] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4770.805438] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4771.946775] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4775.322322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4775.351732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4775.430734] Lustre: Permanently deactivating lustre-OST0000 [ 4775.435458] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4777.264116] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4778.982932] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4780.020220] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4782.326880] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4784.290446] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4786.002603] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4789.274696] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4791.882668] Lustre: Permanently reactivating lustre-OST0000 [ 4791.887403] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 4791.889334] Lustre: Skipped 2 previous similar messages [ 4795.258884] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4795.261193] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4795.265922] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4795.267858] Lustre: Skipped 1 previous similar message [ 4795.282683] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4795.289521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4801.726265] Lustre: server umount lustre-OST0001 complete [ 4817.376067] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4817.426422] Lustre: server umount lustre-OST0000 complete [ 4825.192370] Lustre: server umount lustre-MDT0000 complete [ 4826.876402] LustreError: 411266:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767328246 with bad export cookie 1417544728811742784 [ 4826.879909] LustreError: 411266:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4826.880572] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4827.028765] Lustre: server umount lustre-MDT0001 complete [ 4831.330524] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4832.297790] Key type lgssc unregistered [ 4832.427335] LNet: 415555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4832.429566] LNetError: 415555:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4832.436387] LNet: Removed LNI 192.168.204.123@tcp [ 4832.739812] Key type .llcrypt unregistered [ 4832.741798] Key type ._llcrypt unregistered [ 4858.495315] Key type ._llcrypt registered [ 4858.496459] Key type .llcrypt registered [ 4858.529944] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4858.846560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4858.894032] alg: No test for adler32 (adler32-zlib) [ 4859.742237] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 4859.820887] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4861.400144] Key type lgssc registered [ 4861.782474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4863.142675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4866.561274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4866.567489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4867.755637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4869.093415] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4873.156258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4873.183515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4873.219306] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4873.230535] Lustre: 420093:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4873.234981] Lustre: Permanently deactivating lustre-MDT0001 [ 4873.240677] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4873.362791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4874.651163] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4876.461809] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4876.525110] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4876.527234] LustreError: 420135:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4876.530022] LustreError: 420135:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4876.532830] LustreError: 420135:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 4877.695351] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4881.007961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4881.118446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4882.990549] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4883.171455] LustreError: 422478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4883.175697] LustreError: 422478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4884.741688] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4888.171808] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4888.235262] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4889.952343] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4890.483126] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4891.639587] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4893.551536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4893.551542] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4897.411124] Lustre: Permanently reactivating lustre-MDT0001 [ 4897.416711] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4897.419167] Lustre: Skipped 2 previous similar messages [ 4899.600139] Lustre: lustre-MDT0001: Client a3017c61-1240-493f-b324-fdcaf7787600 (at 192.168.204.23@tcp) reconnecting [ 4901.635525] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4902.177974] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4903.772719] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4903.847110] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4905.059033] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4905.126475] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4905.930522] Lustre: Permanently deactivating lustre-MDT0001 [ 4905.937384] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4905.939739] Lustre: Skipped 2 previous similar messages [ 4907.010346] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4914.656965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4914.659904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4914.666884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4915.169672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4915.170877] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4915.174247] Lustre: Skipped 1 previous similar message [ 4917.888322] LustreError: 424794:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4917.919727] Lustre: server umount lustre-MDT0000 complete [ 4919.396045] LustreError: 420085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767328338 with bad export cookie 15169813759478581043 [ 4919.400696] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4919.468519] LustreError: 425044:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4919.470220] LustreError: 425044:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4919.519230] Lustre: server umount lustre-MDT0001 complete [ 4926.829114] LustreError: 425244:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4926.830934] LustreError: 425244:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4926.849314] Lustre: server umount lustre-OST0000 complete [ 4930.397049] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:32:30 (1767328350) [ 4938.925670] LustreError: 427141:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4938.927233] LustreError: 427141:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4938.967495] Lustre: server umount lustre-OST0001 complete [ 4943.222698] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 4962.959037] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4968.706283] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4968.989911] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4969.767672] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4970.436704] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4975.464462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4975.488460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4975.569260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4975.572734] Lustre: Skipped 2 previous similar messages [ 4975.581420] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4975.618551] Lustre: lustre-MDT0000: new disk, initializing [ 4975.641649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4975.647460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4976.867944] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4983.051527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4983.073310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4983.106866] Lustre: 434333:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4983.131383] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4983.134348] Lustre: Skipped 1 previous similar message [ 4983.173237] Lustre: lustre-MDT0001: new disk, initializing [ 4983.214412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4983.219050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4984.485747] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4987.564828] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4988.741393] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4993.048302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4993.072078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4993.158811] Lustre: lustre-OST0000: new disk, initializing [ 4993.160904] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4993.187820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4993.190883] Lustre: Skipped 1 previous similar message [ 4995.125389] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4998.465748] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5003.262299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5003.267242] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5003.300420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5004.470241] LustreError: 437368:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5010.632289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5010.656352] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5010.715197] Lustre: lustre-OST0001: new disk, initializing [ 5010.718208] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5010.751693] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5012.289106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5012.291987] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5012.322858] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5012.536591] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5014.560082] LustreError: 437368:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5015.497835] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5016.649922] LustreError: 434339:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5017.004810] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 5026.744085] LustreError: 434339:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5027.812607] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5027.813358] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5027.816442] LustreError: Skipped 1 previous similar message [ 5027.820351] Lustre: Skipped 1 previous similar message [ 5027.820934] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5027.826330] Lustre: Skipped 4 previous similar messages [ 5031.120742] Lustre: lustre-OST0001: Not available for connect from 192.168.204.23@tcp (stopping) [ 5033.391738] LustreError: 438863:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5033.393331] LustreError: 438863:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5033.412024] Lustre: server umount lustre-OST0001 complete [ 5038.052561] LustreError: 436755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5038.056857] LustreError: 436755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5043.175134] LustreError: 437380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5043.180712] LustreError: 437380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5048.291229] LustreError: 436754:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5048.292869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5048.294038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5048.294044] Lustre: Skipped 2 previous similar messages [ 5048.299752] LustreError: 436754:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5048.303518] Lustre: Skipped 2 previous similar messages [ 5051.377172] LustreError: 439087:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5051.378862] LustreError: 439087:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5051.415291] Lustre: server umount lustre-OST0000 complete [ 5053.410022] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5053.412868] Lustre: Skipped 1 previous similar message [ 5058.529044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5058.530732] Lustre: Skipped 4 previous similar messages [ 5067.232105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5067.296218] Lustre: server umount lustre-MDT0000 complete [ 5067.744960] LustreError: 434344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5067.751523] LustreError: 434344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5068.741121] LustreError: 438121:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767328488 with bad export cookie 15169813759478583633 [ 5068.744545] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5068.744961] LustreError: 438121:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5083.104106] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5083.142245] LustreError: 439588:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5083.144994] LustreError: 439588:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5083.200794] Lustre: server umount lustre-MDT0001 complete [ 5087.601813] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5088.611324] Key type lgssc unregistered [ 5088.729421] LNet: 440266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5088.732634] LNetError: 440266:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5088.742269] LNet: Removed LNI 192.168.204.123@tcp [ 5089.050914] Key type .llcrypt unregistered [ 5089.051940] Key type ._llcrypt unregistered [ 5108.514720] Key type ._llcrypt registered [ 5108.517242] Key type .llcrypt registered [ 5108.553799] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 5131.613952] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5131.968044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5132.032068] alg: No test for adler32 (adler32-zlib) [ 5132.879423] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 5132.961523] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5134.544101] Key type lgssc registered [ 5134.897825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5140.859616] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5141.020360] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5141.487442] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5142.222850] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5147.192179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5147.211813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5147.216386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5148.299588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5148.314310] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5148.354271] Lustre: lustre-MDT0000: new disk, initializing [ 5148.380399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5148.387446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5149.607512] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5155.573438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5155.599467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5155.635455] Lustre: 450283:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5155.658656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5155.660477] Lustre: Skipped 1 previous similar message [ 5155.695332] Lustre: lustre-MDT0001: new disk, initializing [ 5155.727845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5155.739958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5155.743484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5156.979528] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5159.982555] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5161.121693] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5165.463215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5165.489618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5165.580533] Lustre: lustre-OST0000: new disk, initializing [ 5165.582394] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5165.611120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5167.452564] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5170.167732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5170.170639] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5170.206372] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5170.444562] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5171.716426] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5171.789299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5173.067338] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5173.138639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5175.267076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5175.267237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5175.268621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5175.269065] LustreError: Skipped 1 previous similar message [ 5175.273427] Lustre: Skipped 1 previous similar message [ 5179.823070] LustreError: 453914:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5179.858282] Lustre: server umount lustre-OST0000 complete [ 5181.408793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5181.411123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5181.416090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5181.417785] Lustre: Skipped 1 previous similar message [ 5185.505196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5185.505880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5187.454567] LustreError: 454164:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5187.456326] LustreError: 454164:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5187.485291] Lustre: server umount lustre-MDT0000 complete [ 5189.006335] LustreError: 450275:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767328608 with bad export cookie 14804783591594282844 [ 5189.010054] LustreError: 450275:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5189.010740] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5189.083475] LustreError: 454415:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5189.085421] LustreError: 454415:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5189.146487] Lustre: server umount lustre-MDT0001 complete [ 5191.728087] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 23:36:51 (1767328611) [ 5194.922516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5195.117142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5196.340428] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5200.282075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5200.352952] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5200.354766] Lustre: Skipped 1 previous similar message [ 5200.405214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5201.718020] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5203.413068] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5204.563413] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5207.822138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5209.728324] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5211.388447] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5261.280868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5261.282916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5261.286650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5265.772093] LustreError: 458008:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5265.773920] LustreError: 458008:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5265.816200] Lustre: server umount lustre-OST0000 complete [ 5267.221323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5272.407652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5272.516227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5272.518569] Lustre: Skipped 1 previous similar message [ 5274.315022] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5276.017297] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5281.767084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.770818] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5286.887397] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.891955] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5311.458617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5311.459957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5311.463305] Lustre: Skipped 2 previous similar messages [ 5311.464836] Lustre: Skipped 2 previous similar messages [ 5315.375084] LustreError: 459862:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5315.376902] LustreError: 459862:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5315.426730] Lustre: server umount lustre-OST0000 complete [ 5318.112765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5318.115255] LustreError: Skipped 1 previous similar message [ 5321.698660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5321.707250] Lustre: Skipped 1 previous similar message [ 5323.237034] Lustre: server umount lustre-MDT0000 complete [ 5324.715042] LustreError: 455303:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767328744 with bad export cookie 14804783591594283726 [ 5324.718500] LustreError: 455303:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5324.719217] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5324.790203] LustreError: 460362:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5324.792110] LustreError: 460362:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5324.844742] Lustre: server umount lustre-MDT0001 complete [ 5329.023332] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5330.034636] Key type lgssc unregistered [ 5330.156345] LNet: 461037:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5330.158814] LNetError: 461037:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5330.165395] LNet: Removed LNI 192.168.204.123@tcp [ 5330.467614] Key type .llcrypt unregistered [ 5330.468633] Key type ._llcrypt unregistered [ 5357.961803] Key type ._llcrypt registered [ 5357.962959] Key type .llcrypt registered [ 5358.000203] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5358.289622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5358.308640] alg: No test for adler32 (adler32-zlib) [ 5359.162190] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 5359.244068] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5360.824157] Key type lgssc registered [ 5361.180724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5363.744284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5363.751152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5364.903026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5366.134343] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5369.919327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5370.050438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5371.325571] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5372.997087] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5374.133993] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5377.429382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5377.543279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5379.368204] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5381.117379] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5389.863201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5400.035109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5400.035327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5400.036465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5400.042378] Lustre: Skipped 1 previous similar message [ 5404.718536] LustreError: 468924:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5404.759518] Lustre: server umount lustre-OST0000 complete [ 5410.274181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5410.275211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5410.279360] Lustre: Skipped 1 previous similar message [ 5410.281301] Lustre: Skipped 1 previous similar message [ 5412.419074] LustreError: 469174:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5412.420924] LustreError: 469174:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5412.449522] Lustre: server umount lustre-MDT0000 complete [ 5413.995039] LustreError: 465482:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767328833 with bad export cookie 17373868192719047635 [ 5413.998774] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5413.999902] LustreError: 465482:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5414.071713] LustreError: 469424:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5414.073573] LustreError: 469424:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5414.127521] Lustre: server umount lustre-MDT0001 complete [ 5418.635372] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5419.624267] Key type lgssc unregistered [ 5419.750355] LNet: 470100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5419.752405] LNetError: 470100:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5419.761330] LNet: Removed LNI 192.168.204.123@tcp [ 5420.080379] Key type .llcrypt unregistered [ 5420.081639] Key type ._llcrypt unregistered [ 5446.274339] Key type ._llcrypt registered [ 5446.275343] Key type .llcrypt registered [ 5446.313092] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5446.679833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5446.720430] alg: No test for adler32 (adler32-zlib) [ 5447.567436] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 5447.648851] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5449.224146] Key type lgssc registered [ 5449.587095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5452.141336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5452.148974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5453.303992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5454.541441] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5458.553951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5458.675381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5460.036610] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5461.841161] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5463.005409] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5466.308701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5466.429653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5468.241627] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5469.994339] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5477.740033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5488.099225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5488.099839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5488.104256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5488.109023] Lustre: Skipped 1 previous similar message [ 5490.158653] LustreError: 477147:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5490.197478] Lustre: server umount lustre-OST0000 complete [ 5493.217539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5493.221112] Lustre: Skipped 1 previous similar message [ 5493.223062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5493.225463] Lustre: Skipped 1 previous similar message [ 5498.336979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5498.338657] Lustre: Skipped 1 previous similar message [ 5503.457163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5503.459026] Lustre: Skipped 1 previous similar message [ 5506.016105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5506.047081] LustreError: 477397:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5506.048696] LustreError: 477397:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5506.073119] Lustre: server umount lustre-MDT0000 complete [ 5507.541039] LustreError: 477398:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767328926 with bad export cookie 9432306627511711244 [ 5507.547524] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5507.547860] LustreError: 477398:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5507.627231] LustreError: 477648:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5507.629482] LustreError: 477648:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5507.681593] Lustre: server umount lustre-MDT0001 complete [ 5511.908127] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5512.914513] Key type lgssc unregistered [ 5513.035356] LNet: 478323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5513.037719] LNetError: 478323:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5513.046307] LNet: Removed LNI 192.168.204.123@tcp [ 5513.318819] Key type .llcrypt unregistered [ 5513.319812] Key type ._llcrypt unregistered [ 5543.546813] Key type ._llcrypt registered [ 5543.547944] Key type .llcrypt registered [ 5543.584313] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5543.915313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5543.953802] alg: No test for adler32 (adler32-zlib) [ 5544.798747] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 5544.877932] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5546.456119] Key type lgssc registered [ 5546.796740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5549.191720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5549.198072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5550.360337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5551.589199] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5555.371277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5555.493696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5556.686560] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5558.297540] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5559.418633] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5562.693251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5562.826574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5564.588137] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5566.224217] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5575.143827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5585.378933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5585.380290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5585.380799] LustreError: Skipped 1 previous similar message [ 5585.383277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5585.383614] Lustre: Skipped 1 previous similar message [ 5589.998110] LustreError: 486194:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5590.033582] Lustre: server umount lustre-OST0000 complete [ 5591.520717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5591.522622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5591.525797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5591.527455] Lustre: Skipped 1 previous similar message [ 5595.617410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5595.618377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5597.570335] LustreError: 486444:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5597.572081] LustreError: 486444:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5597.599385] Lustre: server umount lustre-MDT0000 complete [ 5599.146644] LustreError: 482752:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767329018 with bad export cookie 10468175614561851554 [ 5599.152379] LustreError: 482752:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5599.152412] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5599.229383] LustreError: 486693:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5599.232581] LustreError: 486693:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5599.289710] Lustre: server umount lustre-MDT0001 complete [ 5603.632916] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5604.648908] Key type lgssc unregistered [ 5604.772468] LNet: 487370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5604.774598] LNetError: 487370:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5604.782265] LNet: Removed LNI 192.168.204.123@tcp [ 5605.066349] Key type .llcrypt unregistered [ 5605.067605] Key type ._llcrypt unregistered [ 5629.308152] Key type ._llcrypt registered [ 5629.309313] Key type .llcrypt registered [ 5629.353702] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5629.747537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5629.815761] alg: No test for adler32 (adler32-zlib) [ 5630.654603] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 5630.729923] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5632.304113] Key type lgssc registered [ 5632.634044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5634.986471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5634.992269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5636.160620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5637.379874] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5641.493177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5641.615415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5642.870947] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5644.515982] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5645.579775] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5648.788370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5648.906231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5650.655529] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5652.207125] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5661.219665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5666.274212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5666.276060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5666.277627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5666.279280] LustreError: Skipped 1 previous similar message [ 5668.271504] LustreError: 494455:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5668.307481] Lustre: server umount lustre-OST0000 complete [ 5671.395685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5671.397059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5671.399613] Lustre: Skipped 2 previous similar messages [ 5671.402828] Lustre: Skipped 1 previous similar message [ 5672.257702] LustreError: 494705:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5672.259695] LustreError: 494705:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5672.286955] Lustre: server umount lustre-MDT0000 complete [ 5673.988905] LustreError: 494369:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767329093 with bad export cookie 3842543796471849946 [ 5673.996071] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5673.999825] LustreError: 494369:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5674.086153] LustreError: 494954:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5674.088207] LustreError: 494954:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5674.146183] Lustre: server umount lustre-MDT0001 complete [ 5679.704786] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5680.760986] Key type lgssc unregistered [ 5680.891379] LNet: 495631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5680.894547] LNetError: 495631:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5680.903325] LNet: Removed LNI 192.168.204.123@tcp [ 5681.188257] Key type .llcrypt unregistered [ 5681.189697] Key type ._llcrypt unregistered [ 5685.208923] Key type ._llcrypt registered [ 5685.210372] Key type .llcrypt registered [ 5685.256686] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5697.687206] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 5719.468932] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5719.777398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5719.810317] alg: No test for adler32 (adler32-zlib) [ 5720.657309] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 5720.736985] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5722.312102] Key type lgssc registered [ 5722.628123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5728.553067] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5728.694477] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5729.242667] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5730.287760] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5736.201350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5736.233796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5736.240701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5737.333268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5737.345306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5737.392389] Lustre: lustre-MDT0000: new disk, initializing [ 5737.416898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5737.422815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5738.920357] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5746.165744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5746.186197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5746.218378] Lustre: 504864:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5746.242808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5746.244460] Lustre: Skipped 1 previous similar message [ 5746.282608] Lustre: lustre-MDT0001: new disk, initializing [ 5746.317189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5746.330942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5746.334351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5747.714989] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5751.176478] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5752.670754] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5758.452056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5758.482277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5758.605075] Lustre: lustre-OST0000: new disk, initializing [ 5758.608518] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5758.656427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5761.214950] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5764.960013] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5766.550778] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5766.650100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5766.653154] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5766.687806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5767.687293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5769.010889] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5769.068223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5771.746164] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5771.748099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5771.749146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5771.749465] LustreError: Skipped 1 previous similar message [ 5771.756406] Lustre: Skipped 1 previous similar message [ 5775.790058] LustreError: 508502:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5775.824635] Lustre: server umount lustre-OST0000 complete [ 5777.376788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5777.379364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5777.383306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5777.385829] Lustre: Skipped 1 previous similar message [ 5781.985176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5781.985802] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5783.356715] LustreError: 508751:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5783.358456] LustreError: 508751:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5783.386641] Lustre: server umount lustre-MDT0000 complete [ 5784.796097] LustreError: 505989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767329204 with bad export cookie 1302487105894790662 [ 5784.798821] LustreError: 505989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5784.799554] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5784.859189] LustreError: 509000:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5784.860770] LustreError: 509000:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5784.905370] Lustre: server umount lustre-MDT0001 complete [ 5787.304491] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:46:47 (1767329207) [ 5790.220900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5790.375888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5791.477380] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5795.057930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5795.169161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5796.303367] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5797.867310] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5798.907306] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5801.961291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5803.696233] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5805.231497] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5820.900652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5820.901322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5820.902024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5820.902028] Lustre: Skipped 1 previous similar message [ 5820.902654] LustreError: Skipped 1 previous similar message [ 5820.905384] Lustre: Skipped 1 previous similar message [ 5824.109562] LustreError: 512514:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5824.111157] LustreError: 512514:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5824.142688] Lustre: server umount lustre-OST0000 complete [ 5826.017861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5826.018050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5826.018778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5826.018782] Lustre: Skipped 1 previous similar message [ 5826.020961] Lustre: Skipped 1 previous similar message [ 5831.740039] LustreError: 512764:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5831.741687] LustreError: 512764:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5831.767595] Lustre: server umount lustre-MDT0000 complete [ 5833.181939] LustreError: 509842:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767329252 with bad export cookie 1302487105894791544 [ 5833.184897] LustreError: 509842:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5833.185140] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5833.292987] Lustre: server umount lustre-MDT0001 complete [ 5837.161147] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5838.016541] Key type lgssc unregistered [ 5838.128360] LNet: 513689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5838.130296] LNetError: 513689:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5838.140279] LNet: Removed LNI 192.168.204.123@tcp [ 5838.378658] Key type .llcrypt unregistered [ 5838.379630] Key type ._llcrypt unregistered [ 5851.793278] Key type ._llcrypt registered [ 5851.794202] Key type .llcrypt registered [ 5851.825419] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 5873.296961] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5873.680371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5873.698073] alg: No test for adler32 (adler32-zlib) [ 5874.541132] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 5874.612833] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5876.184139] Key type lgssc registered [ 5876.514365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5881.992048] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5882.148174] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5882.792167] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5883.474904] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5885.532762] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5887.667330] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5892.457050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5892.473955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5892.477733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5893.553722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5893.564093] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5893.598376] Lustre: lustre-MDT0000: new disk, initializing [ 5893.619895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5893.625253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5894.721209] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5900.409104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5900.431023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5900.460547] Lustre: 523360:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5900.480294] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5900.481684] Lustre: Skipped 1 previous similar message [ 5900.514925] Lustre: lustre-MDT0001: new disk, initializing [ 5900.543868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5900.554029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5900.556637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5901.720877] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5904.588231] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5905.679915] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5909.860950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5909.882590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5909.971954] Lustre: lustre-OST2710: new disk, initializing [ 5909.973569] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5909.999637] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5911.687138] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5914.543530] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 5916.156490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5916.161212] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5916.199173] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5918.676428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5918.695545] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5918.748744] Lustre: lustre-OST03e8: new disk, initializing [ 5918.750365] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5918.779435] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5920.473483] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5922.131029] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 5924.571473] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5924.634498] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5924.861949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5924.865323] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5924.898419] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5925.901112] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5925.963650] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5927.210535] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5927.274952] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5928.455798] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5928.523850] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5929.955290] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5929.956744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5929.960616] Lustre: Skipped 2 previous similar messages [ 5929.962255] Lustre: Skipped 1 previous similar message [ 5935.073302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5935.076072] Lustre: Skipped 5 previous similar messages [ 5935.549748] LustreError: 528736:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5935.580735] Lustre: server umount lustre-MDT0000 complete [ 5936.749905] LustreError: 524485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767329356 with bad export cookie 2708965664742322573 [ 5936.752691] LustreError: 524485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5936.753459] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5936.815112] LustreError: 528937:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5936.816641] LustreError: 528937:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5936.861716] Lustre: server umount lustre-MDT0001 complete [ 5949.427189] LustreError: 529281:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5949.428989] LustreError: 529281:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5949.446634] Lustre: server umount lustre-OST2710 complete [ 5959.729980] LustreError: 529483:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5959.731869] LustreError: 529483:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5959.769671] Lustre: server umount lustre-OST03e8 complete [ 5973.010662] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 5991.668085] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5997.387048] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5997.551779] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5998.060404] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5998.806283] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6003.517087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6003.537244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6003.613302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6003.625020] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6003.659300] Lustre: lustre-MDT0000: new disk, initializing [ 6003.680856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6003.686353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6004.840342] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6010.747218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6010.769339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6010.809768] Lustre: 538319:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6010.907708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6012.036663] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6014.860666] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6015.891176] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6020.102899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6020.128573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6020.245730] Lustre: lustre-OST0000: new disk, initializing [ 6020.248474] Lustre: Skipped 1 previous similar message [ 6020.251890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6020.255672] Lustre: Skipped 2 previous similar messages [ 6020.293419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6020.297686] Lustre: Skipped 1 previous similar message [ 6021.917260] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6025.078694] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6026.841607] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6030.335169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6030.338293] Lustre: Skipped 1 previous similar message [ 6030.340461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6030.371781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6031.004536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 6032.105871] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6032.167736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6035.430432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6035.431713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6035.432955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6035.438143] Lustre: Skipped 2 previous similar messages [ 6035.449191] LustreError: Skipped 1 previous similar message [ 6038.835130] LustreError: 541976:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6038.838819] LustreError: 541976:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6038.894762] Lustre: server umount lustre-OST0000 complete [ 6041.568777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6041.570797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6041.574326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6041.577378] Lustre: Skipped 1 previous similar message [ 6045.665699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6045.666652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6054.881303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6054.882809] Lustre: Skipped 4 previous similar messages [ 6055.392082] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6055.430920] LustreError: 542225:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6055.432595] LustreError: 542225:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6055.457659] Lustre: server umount lustre-MDT0000 complete [ 6057.410187] LustreError: 538311:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767329476 with bad export cookie 2708965664742324232 [ 6057.412695] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6057.413270] LustreError: 538311:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6057.529236] Lustre: server umount lustre-MDT0001 complete [ 6061.424393] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 23:51:20 (1767329480) [ 6062.105289] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6062.666859] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 23:51:22 (1767329482) [ 6063.615569] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6067.559844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6069.565529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6071.374381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6073.198525] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6073.924525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6078.871748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6078.960573] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6078.968440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6079.025540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6080.238305] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6084.605614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6084.658441] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6084.672821] Lustre: 545182:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6086.310332] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6088.380733] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6089.950439] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6093.908866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6093.950680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6094.020235] Lustre: Denying initial registration attempt from nid 192.168.204.123@tcp, specified as failover [ 6094.022490] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6094.025103] LustreError: 547182:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 6094.027219] LustreError: 547182:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 6094.028777] LustreError: 547182:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6094.047388] Lustre: server umount lustre-OST0000 complete [ 6094.048525] LustreError: 547182:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6100.449313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6100.452159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6100.457296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6111.200145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6111.237150] LustreError: 547597:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6111.240777] LustreError: 547597:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6111.268466] Lustre: server umount lustre-MDT0000 complete [ 6112.976184] LustreError: 547600:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767329532 with bad export cookie 2708965664742325100 [ 6112.979292] LustreError: 547600:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6112.979540] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6113.088815] Lustre: server umount lustre-MDT0001 complete [ 6117.974146] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 6119.120700] Key type lgssc unregistered [ 6119.244579] LNet: 548526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6119.248898] LNetError: 548526:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6119.258516] LNet: Removed LNI 192.168.204.123@tcp [ 6119.601893] Key type .llcrypt unregistered [ 6119.603802] Key type ._llcrypt unregistered [ 6124.201432] Key type ._llcrypt registered [ 6124.202855] Key type .llcrypt registered [ 6124.271257] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6124.617129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6124.625262] alg: No test for adler32 (adler32-zlib) [ 6125.487156] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 6125.573877] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 6127.160134] Key type lgssc registered [ 6127.598352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6130.944955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6132.574090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6134.547141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6136.916694] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6137.808290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6169.014455] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6171.952112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6171.957403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6173.082719] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6173.094177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6173.157759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6174.515806] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6179.353221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6179.384827] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6179.394826] Lustre: 554717:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6179.482970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6180.650227] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6182.586994] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6183.603096] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6188.422376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6188.456311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6188.542110] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6188.604418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6190.480420] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6192.239996] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6194.662459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6194.663294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6194.667521] LustreError: Skipped 1 previous similar message [ 6194.677197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6194.682868] Lustre: Skipped 1 previous similar message [ 6199.090183] LustreError: 557384:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6199.153536] Lustre: server umount lustre-OST0000 complete [ 6204.897594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6204.898105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6204.902541] Lustre: Skipped 2 previous similar messages [ 6207.228797] LustreError: 557635:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6207.230632] LustreError: 557635:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6207.255693] Lustre: server umount lustre-MDT0000 complete [ 6208.731054] LustreError: 554709:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767329628 with bad export cookie 5939477785458729838 [ 6208.733468] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6208.734017] LustreError: 554709:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6208.795967] LustreError: 557886:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6208.797721] LustreError: 557886:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6208.842962] Lustre: server umount lustre-MDT0001 complete [ 6213.006684] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 6213.895444] Key type lgssc unregistered [ 6214.019287] LNet: 558561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6214.021425] LNetError: 558561:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6214.028324] LNet: Removed LNI 192.168.204.123@tcp [ 6214.271454] Key type .llcrypt unregistered [ 6214.272513] Key type ._llcrypt unregistered [ 6243.112839] Key type ._llcrypt registered [ 6243.114554] Key type .llcrypt registered [ 6243.166390] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6243.479316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6243.515088] alg: No test for adler32 (adler32-zlib) [ 6244.367788] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 6244.453331] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 6246.040103] Key type lgssc registered [ 6246.416855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6249.460518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6249.467408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6250.679365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6251.908955] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6255.446379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6255.596839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6256.969212] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6258.546773] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6259.576050] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6263.371387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6263.476530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6265.863789] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6268.303285] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6281.185151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6281.187068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6281.190349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6283.983024] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 6285.282561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6285.286359] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6287.051442] LustreError: 565997:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6287.083217] Lustre: server umount lustre-MDT0000 complete [ 6289.098921] LustreError: 564884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6289.109808] LustreError: 564884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6289.170748] LustreError: 563278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767329708 with bad export cookie 13197286273597899234 [ 6289.177203] LustreError: 563278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6289.177229] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6290.400858] LustreError: 564884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6290.401153] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6290.410480] Lustre: Skipped 1 previous similar message [ 6290.412983] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6290.414438] Lustre: Skipped 2 previous similar messages [ 6294.218657] LustreError: 564884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6294.433587] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6294.435850] Lustre: Skipped 1 previous similar message [ 6295.313660] LustreError: 566249:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6295.317353] LustreError: 566249:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6295.383597] Lustre: server umount lustre-MDT0001 complete [ 6298.421685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6303.533870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6313.952592] LustreError: 562606:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000932a807f x1853179468734336/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6314.055096] LustreError: 565289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6314.060639] LustreError: 565289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6314.078422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6314.098987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 6315.203177] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6319.203552] LustreError: 565289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6319.213523] LustreError: 565289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6319.775528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6319.986328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6321.622958] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6323.506438] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6324.071485] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6330.214349] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 6331.647614] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6332.243771] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6334.439239] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6334.440246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6334.443784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6334.447815] Lustre: Skipped 1 previous similar message [ 6348.256124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6348.285189] LustreError: 569081:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6348.287367] LustreError: 569081:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6348.347773] Lustre: server umount lustre-OST0000 complete [ 6350.817344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6350.819526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6350.822996] Lustre: Skipped 1 previous similar message [ 6350.824783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6350.828849] Lustre: Skipped 4 previous similar messages [ 6356.308501] LustreError: 569335:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6356.312535] LustreError: 569335:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6356.349321] Lustre: server umount lustre-MDT0000 complete [ 6358.426188] LustreError: 563276:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767329777 with bad export cookie 13197286273597906388 [ 6358.432592] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6358.593456] Lustre: server umount lustre-MDT0001 complete [ 6363.814191] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 6365.485279] Key type lgssc unregistered [ 6365.654781] LNet: 570260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6365.659896] LNetError: 570260:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6365.672728] LNet: Removed LNI 192.168.204.123@tcp [ 6366.162542] Key type .llcrypt unregistered [ 6366.164864] Key type ._llcrypt unregistered [ 6401.083236] Key type ._llcrypt registered [ 6401.084602] Key type .llcrypt registered [ 6401.120938] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6401.519643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6401.530636] alg: No test for adler32 (adler32-zlib) [ 6402.411630] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 6402.511150] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 6404.120155] Key type lgssc registered [ 6404.697914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6408.497268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6408.510092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6409.697554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6411.136200] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6416.907682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6417.055299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6418.799954] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6421.471713] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6423.252616] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6426.787763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6426.789061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6426.794404] Lustre: Skipped 1 previous similar message [ 6432.225737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6432.228548] Lustre: Skipped 2 previous similar messages [ 6437.345603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6437.348703] Lustre: Skipped 1 previous similar message [ 6438.880140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6438.924295] LustreError: 576438:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6438.954865] Lustre: server umount lustre-MDT0000 complete [ 6440.492930] LustreError: 575766:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767329859 with bad export cookie 9850463152451334027 [ 6440.499436] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6440.499437] LustreError: 575766:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6440.590074] LustreError: 576642:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6440.592109] LustreError: 576642:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6440.644965] Lustre: server umount lustre-MDT0001 complete [ 6453.003377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6453.135846] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6453.150482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6453.267949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6455.069056] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6460.820411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6460.875281] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6460.893086] Lustre: 578036:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6461.024962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6462.640220] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6465.030203] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6466.655535] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6471.891803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6472.002516] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6472.009767] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6472.016523] LustreError: 580083:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 6472.021363] LustreError: 580083:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 6472.025378] LustreError: 580083:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6472.069620] Lustre: server umount lustre-OST0000 complete [ 6472.072170] LustreError: 580083:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6477.907574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6478.000253] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6478.114088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6480.742578] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6483.366302] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6483.518089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 6489.803718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6489.838899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6489.875085] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6489.928560] Lustre: lustre-OST0001: new disk, initializing [ 6489.930950] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6489.977978] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6492.358526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6492.362114] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6492.393287] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6492.443333] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6494.795529] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6497.765581] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6497.770120] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6497.776459] Lustre: Skipped 1 previous similar message [ 6497.777699] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6497.782169] Lustre: Skipped 1 previous similar message [ 6498.100125] LustreError: 582738:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6498.105127] LustreError: 582738:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6498.136941] Lustre: server umount lustre-OST0001 complete [ 6502.887544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6502.888185] LustreError: 581306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6502.894284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6502.895345] Lustre: Skipped 1 previous similar message [ 6502.903867] LustreError: 581306:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6508.012521] LustreError: 581306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6508.024587] LustreError: 581306:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6512.041425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6512.043631] LustreError: 580743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6512.044882] Lustre: Skipped 4 previous similar messages [ 6512.053393] LustreError: 580743:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6514.144198] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6514.173121] LustreError: 582939:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6514.175131] LustreError: 582939:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6514.218859] Lustre: server umount lustre-OST0000 complete [ 6517.221658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6517.222936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6517.228064] LustreError: Skipped 1 previous similar message [ 6522.505642] LustreError: 583191:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6522.507874] LustreError: 583191:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6522.542498] Lustre: server umount lustre-MDT0000 complete [ 6524.456982] LustreError: 578028:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767329943 with bad export cookie 9850463152451334580 [ 6524.463713] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6524.466727] LustreError: 578028:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6524.639491] Lustre: server umount lustre-MDT0001 complete [ 6529.971080] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 6531.297140] Key type lgssc unregistered [ 6531.444643] LNet: 584118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6531.449318] LNetError: 584118:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6531.459561] LNet: Removed LNI 192.168.204.123@tcp [ 6531.895710] Key type .llcrypt unregistered [ 6531.897327] Key type ._llcrypt unregistered [ 6554.393172] Key type ._llcrypt registered [ 6554.394685] Key type .llcrypt registered [ 6554.428229] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 6579.621842] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6580.017729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6580.027221] alg: No test for adler32 (adler32-zlib) [ 6580.873803] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 6580.952333] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 6582.528171] Key type lgssc registered [ 6582.852730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6588.844709] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6589.125181] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6589.864867] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6590.855548] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6597.041150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6597.071773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6597.078204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6598.176798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6598.187272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6598.224024] Lustre: lustre-MDT0000: new disk, initializing [ 6598.249541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6598.256084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6599.489551] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6606.122376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6606.157881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6606.201579] Lustre: 594139:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6606.243526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6606.246261] Lustre: Skipped 1 previous similar message [ 6606.283887] Lustre: lustre-MDT0001: new disk, initializing [ 6606.332245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6606.353452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6606.359703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6607.908267] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6610.978025] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6612.389738] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6616.889163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6616.928153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6617.048120] Lustre: lustre-OST0000: new disk, initializing [ 6617.051133] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6617.089752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6617.242944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6617.247221] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6617.284520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6619.094988] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6622.404707] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6623.989143] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6624.070787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6625.292324] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6625.362787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6627.812031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6627.812120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6627.813929] LustreError: Skipped 1 previous similar message [ 6627.815238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6627.818951] Lustre: Skipped 1 previous similar message [ 6627.822491] Lustre: Skipped 1 previous similar message [ 6628.020225] LustreError: 597770:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6628.075491] Lustre: server umount lustre-OST0000 complete [ 6631.905525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6631.908124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6631.912762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6632.930837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6632.931179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6632.938334] Lustre: Skipped 1 previous similar message [ 6638.050874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6638.055614] Lustre: Skipped 1 previous similar message [ 6643.170411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6643.174393] Lustre: Skipped 1 previous similar message [ 6644.192200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6644.229068] LustreError: 598019:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6644.230848] LustreError: 598019:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6644.274350] Lustre: server umount lustre-MDT0000 complete [ 6646.177518] LustreError: 595265:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767330065 with bad export cookie 5145437102265549114 [ 6646.180514] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6646.183439] LustreError: 595265:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6646.266404] LustreError: 598283:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6646.268616] LustreError: 598283:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6646.348955] Lustre: server umount lustre-MDT0001 complete [ 6649.958925] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 00:01:09 (1767330069) [ 6654.920723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6658.722858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6676.880439] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 6705.685669] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6712.059590] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6712.523509] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6713.422387] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6714.441657] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6721.971075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6722.008204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6722.138961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6722.158113] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6722.215662] Lustre: lustre-MDT0000: new disk, initializing [ 6722.244321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6722.254272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6724.022932] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6732.074243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6732.112919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6732.164601] Lustre: 608428:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6732.203429] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6732.206789] Lustre: Skipped 1 previous similar message [ 6732.259024] Lustre: lustre-MDT0001: new disk, initializing [ 6732.307366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6732.328770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6732.336598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6733.901102] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6737.333178] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6738.590706] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6744.816129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6744.859042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6745.008444] Lustre: lustre-OST0000: new disk, initializing [ 6745.011963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6745.058584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6747.579176] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6751.492948] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6753.291589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6753.297045] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6753.337247] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6753.345704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6753.418396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6755.104674] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6755.228339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6758.372784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6758.374032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6758.378709] LustreError: Skipped 1 previous similar message [ 6758.386673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6758.392762] Lustre: Skipped 1 previous similar message [ 6762.225258] LustreError: 612057:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6762.229167] LustreError: 612057:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6762.291463] Lustre: server umount lustre-OST0000 complete [ 6768.097402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6768.102399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6768.110083] Lustre: Skipped 1 previous similar message [ 6770.641205] LustreError: 612307:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6770.645341] LustreError: 612307:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6770.684131] Lustre: server umount lustre-MDT0000 complete [ 6772.912133] LustreError: 608420:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767330192 with bad export cookie 5145437102265549996 [ 6772.919255] LustreError: 608420:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6772.919822] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6773.072881] Lustre: server umount lustre-MDT0001 complete [ 6776.977916] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:03:16 (1767330196) [ 6781.085950] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:03:20 (1767330200) [ 6787.638943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6787.898262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6789.633217] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6793.186066] LustreError: 613926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6793.195350] LustreError: 613926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6795.003340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6796.778122] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6799.269225] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6801.023424] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6805.799464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6808.431715] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6810.346303] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6824.564832] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6825.450185] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6826.315135] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6827.190226] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6828.116444] Lustre: Failing over lustre-MDT0000 [ 6828.252354] LustreError: 616751:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6828.254608] LustreError: 616751:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6828.279898] Lustre: server umount lustre-MDT0000 complete [ 6830.564545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6830.567370] LustreError: 613904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6830.569857] Lustre: Skipped 3 previous similar messages [ 6830.579128] LustreError: 613904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6832.344811] LustreError: 613903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6832.354545] LustreError: 613903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6835.683191] LustreError: 613907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6835.696607] LustreError: 613907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6840.802312] LustreError: 613907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6840.810769] LustreError: 613907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6845.920236] Lustre: 590613:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767330249/real 1767330249] req@0000000021413e63 x1853179822168064/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767330265 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6845.934975] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6847.330597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6851.042459] LustreError: 613926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6851.051571] LustreError: 613926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6856.165408] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x476843c866a74c38 [ 6856.168273] Lustre: MGC192.168.204.123@tcp: Connection restored to 0@lo (at 0@lo) [ 6856.253814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6856.255673] Lustre: Skipped 2 previous similar messages [ 6856.286769] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6857.620426] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6857.933390] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6861.292947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6861.302921] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6861.328331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6863.022343] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6863.887491] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6865.990763] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6866.733942] Lustre: Failing over lustre-MDT0000 [ 6866.856979] LustreError: 618412:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6866.858564] LustreError: 618412:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6866.882593] Lustre: server umount lustre-MDT0000 complete [ 6868.182126] LustreError: 616448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6868.185875] LustreError: 616448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6871.525769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6871.526897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6871.532966] Lustre: Skipped 2 previous similar messages [ 6883.000080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6883.039847] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6883.151148] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6883.529807] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6884.389995] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6888.438514] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6888.443031] Lustre: Skipped 2 previous similar messages [ 6888.453174] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6888.483373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6890.238456] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6890.889026] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6893.538949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6893.539730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6893.543524] Lustre: Skipped 4 previous similar messages [ 6899.404283] LustreError: 620019:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6899.407547] LustreError: 620019:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6899.437128] Lustre: server umount lustre-MDT0000 complete [ 6901.039049] LustreError: 618413:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767330320 with bad export cookie 5145437102265552691 [ 6901.046111] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6901.046920] LustreError: 618413:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6901.185393] Lustre: server umount lustre-MDT0001 complete [ 6914.530116] Lustre: server umount lustre-OST0000 complete [ 6923.098764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6923.245838] LustreError: 621970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6923.251920] LustreError: 621970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 6923.272995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6923.275488] Lustre: Skipped 1 previous similar message [ 6924.419503] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6928.236642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6929.566328] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6931.333389] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6932.513652] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6936.088132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6937.870700] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6939.761611] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6948.519077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6953.960568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6953.962420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6953.964643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6953.964653] Lustre: Skipped 3 previous similar messages [ 6953.970586] Lustre: Skipped 4 previous similar messages [ 6953.974859] LustreError: Skipped 1 previous similar message [ 6957.348651] Lustre: server umount lustre-OST0000 complete [ 6965.844128] LustreError: 624870:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6965.848936] LustreError: 624870:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 6965.884769] Lustre: server umount lustre-MDT0000 complete [ 6968.178504] LustreError: 624871:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767330387 with bad export cookie 5145437102265553426 [ 6968.184398] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6968.186207] LustreError: 624871:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6968.333559] Lustre: server umount lustre-MDT0001 complete [ 6974.473055] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 6975.502294] Key type lgssc unregistered [ 6975.651549] LNet: 625796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6975.655127] LNetError: 625796:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6975.665523] LNet: Removed LNI 192.168.204.123@tcp [ 6976.040899] Key type .llcrypt unregistered [ 6976.042595] Key type ._llcrypt unregistered [ 7015.097583] Key type ._llcrypt registered [ 7015.098846] Key type .llcrypt registered [ 7015.144767] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7015.503197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7015.536429] alg: No test for adler32 (adler32-zlib) [ 7016.396001] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 7016.478172] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7018.064089] Key type lgssc registered [ 7018.359716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7021.044457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7021.050894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7022.214631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7023.510070] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7028.435470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7028.543641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7029.946304] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7032.482486] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7034.049740] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7038.679573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7038.852289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7041.385320] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7043.942685] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7047.085741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7059.425456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7059.430075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7059.437836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7063.524624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7063.524826] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7063.528989] Lustre: Skipped 1 previous similar message [ 7063.537368] Lustre: Skipped 1 previous similar message [ 7065.103146] LustreError: 633144:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7065.141359] Lustre: server umount lustre-MDT0000 complete [ 7066.842832] LustreError: 630424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767330486 with bad export cookie 16501260285785321105 [ 7066.848661] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7066.850449] LustreError: 630424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7066.924482] LustreError: 633346:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7066.926124] LustreError: 633346:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7066.965972] Lustre: server umount lustre-MDT0001 complete [ 7078.770370] LustreError: 633691:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7078.771985] LustreError: 633691:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7078.807447] Lustre: server umount lustre-OST0000 complete [ 7085.157270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7085.291686] LustreError: 634902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7085.297950] LustreError: 634902:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7085.310932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7086.411687] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7090.103505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7090.209826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7091.357923] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7092.956752] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7094.120924] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7097.375914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7099.253620] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7100.839669] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7110.822075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7120.864747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7120.866691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7120.869940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7120.871493] Lustre: Skipped 1 previous similar message [ 7124.154709] LustreError: 637655:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7124.156794] LustreError: 637655:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7124.178577] Lustre: server umount lustre-MDT0000 complete [ 7125.289157] LustreError: 634926:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767330544 with bad export cookie 16501260285785322204 [ 7125.291499] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7125.295047] LustreError: 634926:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7125.406717] Lustre: server umount lustre-MDT0001 complete [ 7137.587378] LustreError: 638200:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7137.590348] LustreError: 638200:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7137.639450] Lustre: server umount lustre-OST0000 complete [ 7148.790616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7148.945203] LustreError: 639606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7148.970714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7148.973355] Lustre: Skipped 1 previous similar message [ 7150.626306] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7154.146694] LustreError: 639624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7155.255182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7156.606164] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7158.746353] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7160.033533] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7163.345836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7165.076966] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7166.678852] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7174.703066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7179.747835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7179.749728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7179.751069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7179.751079] Lustre: Skipped 3 previous similar messages [ 7179.767437] Lustre: Skipped 3 previous similar messages [ 7181.043979] LustreError: 642256:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7181.047701] LustreError: 642256:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7181.103765] Lustre: server umount lustre-OST0000 complete [ 7184.866770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7184.868585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7184.874046] Lustre: Skipped 1 previous similar message [ 7184.877485] Lustre: Skipped 1 previous similar message [ 7189.484725] Lustre: server umount lustre-MDT0000 complete [ 7189.986805] LustreError: 639605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7189.995328] LustreError: 639605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7191.641905] LustreError: 640516:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767330610 with bad export cookie 16501260285785323212 [ 7191.648397] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7191.648965] LustreError: 640516:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7191.815714] Lustre: server umount lustre-MDT0001 complete [ 7197.420435] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7198.774341] Key type lgssc unregistered [ 7198.948774] LNet: 643432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7198.953350] LNetError: 643432:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7198.962703] LNet: Removed LNI 192.168.204.123@tcp [ 7199.443445] Key type .llcrypt unregistered [ 7199.445401] Key type ._llcrypt unregistered [ 7239.145614] Key type ._llcrypt registered [ 7239.147579] Key type .llcrypt registered [ 7239.211907] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7239.535754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7239.566908] alg: No test for adler32 (adler32-zlib) [ 7240.462274] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 7240.558282] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7242.152172] Key type lgssc registered [ 7242.775968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7246.750701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7246.762485] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7246.768162] LustreError: 648002:0:(osd_handler.c:8561:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7246.776664] LustreError: 648002:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7246.781108] LustreError: 648002:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7246.785699] LustreError: 648002:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7246.790905] LustreError: 648002:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7252.271419] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7252.276430] LustreError: 648592:0:(osd_handler.c:8561:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7252.289963] LustreError: 648592:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7252.294384] LustreError: 648592:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7252.298975] LustreError: 648592:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7252.304138] LustreError: 648592:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7263.557404] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7265.213595] Key type lgssc unregistered [ 7265.387597] LNet: 649817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7265.392134] LNetError: 649817:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7265.400569] LNet: Removed LNI 192.168.204.123@tcp [ 7265.860163] Key type .llcrypt unregistered [ 7265.862251] Key type ._llcrypt unregistered [ 7282.876836] Key type ._llcrypt registered [ 7282.878372] Key type .llcrypt registered [ 7282.922845] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 7315.990508] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7316.548562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7316.555940] alg: No test for adler32 (adler32-zlib) [ 7317.437421] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 7317.539225] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7319.136256] Key type lgssc registered [ 7319.782036] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7326.612322] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7326.991373] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7327.530096] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7329.173679] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7336.650347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7336.679215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7336.688431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7337.826172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7337.846990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7337.911513] Lustre: lustre-MDT0000: new disk, initializing [ 7337.955506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7337.965950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7339.972553] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7348.835082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7348.873405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7348.931469] Lustre: 658629:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7348.976752] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7348.980446] Lustre: Skipped 1 previous similar message [ 7349.024425] Lustre: lustre-MDT0001: new disk, initializing [ 7349.082326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7349.106376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7349.113368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7350.974520] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7354.982785] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7356.710987] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7363.358201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7363.404276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7363.578254] Lustre: lustre-OST0000: new disk, initializing [ 7363.582464] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7363.643274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7366.383660] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7370.448111] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7372.373688] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7373.847548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7373.854562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7373.920676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7374.571523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 7376.472144] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7376.585432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7378.922918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7378.923766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7378.927792] LustreError: Skipped 1 previous similar message [ 7378.936980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7378.941890] Lustre: Skipped 1 previous similar message [ 7383.604162] LustreError: 662274:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7383.668710] Lustre: server umount lustre-OST0000 complete [ 7389.156390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7389.157627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7389.168366] Lustre: Skipped 2 previous similar messages [ 7392.081941] LustreError: 662523:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7392.085523] LustreError: 662523:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7392.134350] Lustre: server umount lustre-MDT0000 complete [ 7394.412387] LustreError: 658621:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767330813 with bad export cookie 16785761612151864470 [ 7394.422610] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7394.522433] LustreError: 662772:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7394.526110] LustreError: 662772:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7394.619782] Lustre: server umount lustre-MDT0001 complete [ 7398.250636] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7399.016933] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:13:38 (1767330818) [ 7403.270693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7403.468808] LustreError: 663684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7403.475455] LustreError: 663684:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7403.496730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7405.201906] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7408.610256] LustreError: 663684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7411.049286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7411.242749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7413.032078] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7415.579383] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7417.304127] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7422.363452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7422.557092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7425.206119] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7427.900617] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7434.125368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7434.172007] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7434.288795] Lustre: lustre-OST0001: new disk, initializing [ 7434.292238] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7436.677877] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7437.008993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7437.015576] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7437.076421] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7440.773912] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7447.257490] Lustre: lustre-OST0001: Not available for connect from 192.168.204.23@tcp (stopping) [ 7447.261697] Lustre: Skipped 1 previous similar message [ 7447.527410] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7447.529416] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7448.947810] LustreError: 667610:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7448.953311] LustreError: 667610:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7448.984798] Lustre: server umount lustre-OST0001 complete [ 7452.647326] LustreError: 665667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7452.657193] LustreError: 665667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7457.765768] LustreError: 665665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7457.775947] LustreError: 665665:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7462.369972] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7462.374350] LustreError: Skipped 1 previous similar message [ 7462.376831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7462.383224] Lustre: Skipped 1 previous similar message [ 7462.386275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7462.390264] Lustre: Skipped 2 previous similar messages [ 7462.882667] LustreError: 665667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7462.886675] LustreError: 665667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7467.761575] LustreError: 667836:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7467.765779] LustreError: 667836:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7467.824213] Lustre: server umount lustre-OST0000 complete [ 7471.011407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7471.012349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7471.018840] Lustre: Skipped 2 previous similar messages [ 7471.025806] Lustre: Skipped 2 previous similar messages [ 7476.365538] LustreError: 668085:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7476.368850] LustreError: 668085:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7476.400278] Lustre: server umount lustre-MDT0000 complete [ 7478.241472] LustreError: 664589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7478.250055] LustreError: 664589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7478.602846] LustreError: 663665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767330897 with bad export cookie 16785761612151865352 [ 7478.609351] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7478.610346] LustreError: 663665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7484.835751] Lustre: server umount lustre-MDT0001 complete [ 7490.897535] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7492.300260] Key type lgssc unregistered [ 7492.468566] LNet: 669012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7492.472982] LNetError: 669012:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7492.482582] LNet: Removed LNI 192.168.204.123@tcp [ 7492.926250] Key type .llcrypt unregistered [ 7492.928275] Key type ._llcrypt unregistered [ 7520.185471] Key type ._llcrypt registered [ 7520.186734] Key type .llcrypt registered [ 7520.231388] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 7551.861894] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7552.447433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7552.456257] alg: No test for adler32 (adler32-zlib) [ 7553.343979] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 7553.473654] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7555.096166] Key type lgssc registered [ 7555.745443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7563.164207] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7563.268277] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7563.816267] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7564.920669] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7571.582903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7571.607679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7571.613101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7572.730710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7572.746459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7572.797338] Lustre: lustre-MDT0000: new disk, initializing [ 7572.826928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7572.839723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7574.454463] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7582.575928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7582.612580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7582.666542] Lustre: 679030:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7582.714197] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7582.719798] Lustre: Skipped 1 previous similar message [ 7582.778153] Lustre: lustre-MDT0001: new disk, initializing [ 7582.831930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7582.857135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7582.864901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7584.557392] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7587.965985] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7589.178694] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7594.514572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7594.556503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7594.701717] Lustre: lustre-OST0000: new disk, initializing [ 7594.705303] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7594.756792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7596.367315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7596.373338] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7596.435908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7597.473646] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7601.608645] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7603.540085] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7603.633217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7605.470356] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7605.573530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7606.756506] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7606.757749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7606.761044] LustreError: Skipped 1 previous similar message [ 7606.767963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7606.772149] Lustre: Skipped 1 previous similar message [ 7611.878019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7611.880739] Lustre: Skipped 1 previous similar message [ 7612.594477] LustreError: 682659:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7612.646852] Lustre: server umount lustre-OST0000 complete [ 7616.995387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7616.996268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7617.000424] Lustre: Skipped 2 previous similar messages [ 7617.006578] Lustre: Skipped 1 previous similar message [ 7620.489513] LustreError: 682908:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7620.492379] LustreError: 682908:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7620.534050] Lustre: server umount lustre-MDT0000 complete [ 7622.421724] LustreError: 679023:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767331041 with bad export cookie 18308467629834999540 [ 7622.425581] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7622.428543] LustreError: 679023:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7622.506991] LustreError: 683159:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7622.508872] LustreError: 683159:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7622.565823] Lustre: server umount lustre-MDT0001 complete [ 7625.729633] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 00:17:25 (1767331045) [ 7631.494931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7637.430960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7637.603960] Lustre: lustre-MDT0000: new disk, initializing [ 7637.615957] LustreError: 684761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7637.621720] LustreError: 684761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7637.646292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7638.942175] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7643.105951] LustreError: 684779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7643.531074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7643.702592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7645.561376] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7648.225869] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7650.003633] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7651.452673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7651.455423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7651.459857] Lustre: Skipped 1 previous similar message [ 7657.617642] LustreError: 686342:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7657.621803] LustreError: 686342:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7657.660460] Lustre: server umount lustre-MDT0000 complete [ 7658.977878] LustreError: 684761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7658.983794] LustreError: 684761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7659.699837] LustreError: 684741:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767331078 with bad export cookie 18308467629835000422 [ 7659.707321] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7659.861122] Lustre: server umount lustre-MDT0001 complete [ 7664.350615] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:18:03 (1767331083) [ 7669.416427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7669.624984] LustreError: 687505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7669.664159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7671.587791] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7674.850182] LustreError: 687505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7677.408578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7679.161536] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7681.646259] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7683.177649] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7688.030572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7688.212067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7688.215840] Lustre: Skipped 1 previous similar message [ 7690.809942] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7693.464509] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7707.291429] Lustre: Permanently deactivating lustre-OST0000 [ 7707.301795] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7717.373784] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7718.268056] LustreError: 690699:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7718.274808] LustreError: 690699:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 7718.279170] LustreError: 690699:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7719.105810] LustreError: 690748:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 7719.112427] LustreError: 690748:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7726.068436] LustreError: 690845:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7726.072404] LustreError: 690845:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7726.133978] Lustre: server umount lustre-OST0000 complete [ 7727.504543] LustreError: 690997:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 7727.511203] LustreError: 690997:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7729.121164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7729.125043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7729.132076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7729.136476] Lustre: Skipped 3 previous similar messages [ 7739.366241] LustreError: 691143:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7739.370944] LustreError: 691143:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7739.407350] Lustre: server umount lustre-MDT0000 complete [ 7740.385902] LustreError: 687505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7740.394305] LustreError: 687505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7741.604030] LustreError: 687486:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767331160 with bad export cookie 18308467629835000961 [ 7741.611232] LustreError: 687486:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7741.611600] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7751.200968] Lustre: server umount lustre-MDT0001 complete [ 7757.793521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7759.388638] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7760.691681] LustreError: 692571:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7763.653757] Lustre: 692910:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.123@tcp [ 7766.331921] LustreError: 693199:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7766.334439] LustreError: 693199:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7766.365057] Lustre: server umount MGS complete [ 7775.128201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7775.328566] LustreError: 694213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7775.336468] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7775.339267] Lustre: Skipped 1 previous similar message [ 7775.364774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7777.051637] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7782.786294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7782.930002] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7784.500988] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7786.589314] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7787.952135] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7788.689842] Lustre: Permanently reactivating lustre-OST0000 [ 7788.697763] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7788.701628] Lustre: Skipped 2 previous similar messages [ 7811.553141] LustreError: 695864:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7816.567742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7819.109099] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7821.356176] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7831.012195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7831.012306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7831.018303] Lustre: Skipped 1 previous similar message [ 7831.019874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7831.029312] Lustre: Skipped 5 previous similar messages [ 7834.226169] LustreError: 697766:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7834.230588] LustreError: 697766:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7834.290317] Lustre: server umount lustre-OST0000 complete [ 7839.201384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7839.205881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7839.212892] Lustre: Skipped 1 previous similar message [ 7842.739572] Lustre: server umount lustre-MDT0000 complete [ 7844.869262] LustreError: 695754:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767331264 with bad export cookie 18308467629835001920 [ 7844.876807] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7844.878987] LustreError: 695754:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7845.029380] Lustre: server umount lustre-MDT0001 complete [ 7851.238175] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7852.520644] Key type lgssc unregistered [ 7852.674662] LNet: 698945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7852.679415] LNetError: 698945:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7852.690572] LNet: Removed LNI 192.168.204.123@tcp [ 7853.157523] Key type .llcrypt unregistered [ 7853.159798] Key type ._llcrypt unregistered [ 7870.903749] Key type ._llcrypt registered [ 7870.905393] Key type .llcrypt registered [ 7870.965689] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 7901.946604] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7902.438456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7902.443589] alg: No test for adler32 (adler32-zlib) [ 7903.311029] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 7903.400685] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7904.992174] Key type lgssc registered [ 7905.540780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7912.311653] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7912.767684] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7913.497607] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7914.835389] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7922.747774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7922.784402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7922.793666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7923.932346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7923.951346] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7923.994354] Lustre: lustre-MDT0000: new disk, initializing [ 7924.038198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7924.047330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7925.973470] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7934.595676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7934.633222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7934.700793] Lustre: 707755:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7934.750775] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7934.754300] Lustre: Skipped 1 previous similar message [ 7934.807571] Lustre: lustre-MDT0001: new disk, initializing [ 7934.869392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7934.895716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7934.903063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7936.891855] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7940.750655] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7942.219488] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7948.131893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7948.171378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7948.311857] Lustre: lustre-OST0000: new disk, initializing [ 7948.315456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7948.363092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7950.919156] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7955.023874] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7956.942293] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7958.023061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7958.026037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7958.060102] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7959.122678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 7960.927103] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7961.049342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7963.108217] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7963.111054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7963.112646] LustreError: Skipped 1 previous similar message [ 7963.114491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7963.124425] Lustre: Skipped 1 previous similar message [ 7963.134635] Lustre: Skipped 1 previous similar message [ 7968.051189] LustreError: 711399:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7968.112691] Lustre: server umount lustre-OST0000 complete [ 7970.785513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7970.792139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7970.803177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7973.345592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7973.346525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7976.466757] LustreError: 711650:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7976.468993] LustreError: 711650:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7976.499090] Lustre: server umount lustre-MDT0000 complete [ 7978.704605] LustreError: 707747:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767331397 with bad export cookie 2867786655908711465 [ 7978.710985] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7978.711759] LustreError: 707747:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7978.806138] LustreError: 711900:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7978.808111] LustreError: 711900:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7978.879807] Lustre: server umount lustre-MDT0001 complete [ 7983.225422] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7984.115524] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 00:23:23 (1767331403) [ 7990.613931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7990.866741] LustreError: 712953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7990.875103] LustreError: 712953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7990.905148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7992.631902] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7996.387130] LustreError: 712953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7998.674052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7998.804855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8000.420438] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8003.086821] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8004.790432] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8009.811904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8009.991935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8012.511535] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8015.045839] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8015.919537] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8017.610919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8031.714816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8031.719322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8031.727243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8031.731183] Lustre: Skipped 1 previous similar message [ 8034.788895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8036.020124] LustreError: 715799:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8036.023865] LustreError: 715799:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8036.083493] Lustre: server umount lustre-OST0000 complete [ 8039.907477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8039.908358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8039.909598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8039.909606] Lustre: Skipped 2 previous similar messages [ 8039.926181] Lustre: Skipped 2 previous similar messages [ 8044.495753] LustreError: 716048:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8044.499774] LustreError: 716048:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8044.531800] Lustre: server umount lustre-MDT0000 complete [ 8045.025844] LustreError: 712952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8045.034924] LustreError: 712952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8046.799855] LustreError: 714959:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767331466 with bad export cookie 2867786655908712347 [ 8046.805802] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8046.810717] LustreError: 714959:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8046.951199] Lustre: server umount lustre-MDT0001 complete [ 8052.835679] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8054.294943] Key type lgssc unregistered [ 8054.462656] LNet: 716974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8054.467304] LNetError: 716974:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8054.480644] LNet: Removed LNI 192.168.204.123@tcp [ 8054.952260] Key type .llcrypt unregistered [ 8054.955362] Key type ._llcrypt unregistered [ 8066.701018] Key type ._llcrypt registered [ 8066.702188] Key type .llcrypt registered [ 8066.746520] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8066.866384] Key type .llcrypt unregistered [ 8066.867729] Key type ._llcrypt unregistered [ 8099.752666] Key type ._llcrypt registered [ 8099.754172] Key type .llcrypt registered [ 8099.807526] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8100.107490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8100.113378] alg: No test for adler32 (adler32-zlib) [ 8100.971190] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 8101.070599] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 8102.648175] Key type lgssc registered [ 8103.176396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8106.589480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8106.602652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8107.793384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8109.143566] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8114.560692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8114.706591] LustreError: 722437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8114.715122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8114.718501] LustreError: 722437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8117.120459] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8117.795263] LustreError: 722437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8117.805412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8117.808288] LustreError: 722437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8119.410700] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8122.850621] LustreError: 722437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8122.854439] LustreError: 722437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8123.601534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8123.784388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8125.442446] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8134.124493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8134.124583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8134.132390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8134.138589] Lustre: Skipped 1 previous similar message [ 8139.240788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8139.244328] Lustre: Skipped 1 previous similar message [ 8139.445211] LustreError: 724878:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 8139.511435] Lustre: server umount lustre-OST0000 complete [ 8143.844749] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8143.849105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8143.855812] Lustre: Skipped 1 previous similar message [ 8143.859246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8147.919164] LustreError: 725128:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8147.923092] LustreError: 725128:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8147.967714] Lustre: server umount lustre-MDT0000 complete [ 8149.474347] LustreError: 724432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8150.237521] LustreError: 724792:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767331569 with bad export cookie 12935432502590280802 [ 8150.248436] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8150.345937] LustreError: 725378:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8150.350080] LustreError: 725378:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8150.443070] Lustre: server umount lustre-MDT0001 complete [ 8156.456253] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8157.777041] Key type lgssc unregistered [ 8157.915678] LNet: 726053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8157.920162] LNetError: 726053:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8157.928553] LNet: Removed LNI 192.168.204.123@tcp [ 8158.389557] Key type .llcrypt unregistered [ 8158.391854] Key type ._llcrypt unregistered [ 8186.985718] Key type ._llcrypt registered [ 8186.987380] Key type .llcrypt registered [ 8187.026849] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8187.343456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8187.382729] alg: No test for adler32 (adler32-zlib) [ 8188.253724] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 8188.355764] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 8189.952176] Key type lgssc registered [ 8190.549555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8194.307101] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8194.317615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8209.824389] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8214.944209] LustreError: 730474:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.123@tcp: failed processing log, type 4: rc = -110 [ 8241.568251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8243.758410] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8246.177912] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8250.875645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8251.085889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8251.113154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8252.571651] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8256.228952] LustreError: 730501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8256.238339] LustreError: 730501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8257.896226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8258.028504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8259.770370] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8262.434738] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8263.818330] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8266.721373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8266.725779] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8266.733347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8268.265435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8268.265552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8268.265890] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8268.279917] Lustre: Skipped 1 previous similar message [ 8272.627177] LustreError: 733141:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 8272.680741] Lustre: server umount lustre-OST0000 complete [ 8276.964043] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8276.968487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8276.976350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8280.976485] LustreError: 733392:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 8280.980357] LustreError: 733392:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8281.014451] Lustre: server umount lustre-MDT0000 complete [ 8283.165224] LustreError: 730483:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767331702 with bad export cookie 2142361437897207353 [ 8283.172365] LustreError: 730483:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8283.172606] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8283.281094] LustreError: 733643:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8283.284516] LustreError: 733643:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8283.343452] Lustre: server umount lustre-MDT0001 complete [ 8288.932650] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8290.294886] Key type lgssc unregistered [ 8290.432560] LNet: 734318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8290.437251] LNetError: 734318:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8290.445640] LNet: Removed LNI 192.168.204.123@tcp [ 8290.888138] Key type .llcrypt unregistered [ 8290.890442] Key type ._llcrypt unregistered [ 8328.251316] Key type ._llcrypt registered [ 8328.252735] Key type .llcrypt registered [ 8328.311928] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8328.636836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8328.672236] alg: No test for adler32 (adler32-zlib) [ 8329.532285] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 8329.633449] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 8331.224149] Key type lgssc registered [ 8331.817418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8335.585261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8335.592823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8336.772642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8338.069697] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8343.173387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8343.352171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8344.976438] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8347.528088] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8349.210176] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8353.715713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8353.872592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8356.402671] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8358.961819] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8362.093310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8367.587910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8367.590549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8367.596033] Lustre: Skipped 2 previous similar messages [ 8367.599934] Lustre: Skipped 1 previous similar message [ 8368.610409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8368.614177] Lustre: Skipped 1 previous similar message [ 8370.390270] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 8372.751459] LustreError: 741447:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8372.791262] Lustre: server umount lustre-MDT0000 complete [ 8373.730983] LustreError: 738764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8373.741579] LustreError: 738764:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8373.863135] LustreError: 738749:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.23@tcp arrived at 1767331793 with bad export cookie 16976809180416765493 [ 8373.870903] LustreError: 738749:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8374.794797] LustreError: 738749:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767331794 with bad export cookie 16976809180416764849 [ 8374.802497] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8378.849987] LustreError: 739688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8378.850971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8378.860550] LustreError: 739688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8378.868680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8379.873575] LustreError: 739688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8379.883677] LustreError: 739688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8380.913237] LustreError: 741651:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8380.919343] LustreError: 741651:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8380.966072] Lustre: server umount lustre-OST0000 complete [ 8384.952344] LustreError: 739675:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000056bbd6bb x1853181655017216/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8384.978304] LustreError: 742100:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8384.982474] LustreError: 742100:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8385.052917] Lustre: server umount lustre-MDT0001 complete [ 8390.558242] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8391.904818] Key type lgssc unregistered [ 8392.045539] LNet: 742775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8392.049536] LNetError: 742775:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8392.060507] LNet: Removed LNI 192.168.204.123@tcp [ 8392.480958] Key type .llcrypt unregistered [ 8392.483953] Key type ._llcrypt unregistered [ 8428.513174] Key type ._llcrypt registered [ 8428.514604] Key type .llcrypt registered [ 8428.570562] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8428.949834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8428.968376] alg: No test for adler32 (adler32-zlib) [ 8429.831997] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 8429.917983] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 8431.488156] Key type lgssc registered [ 8431.957659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8435.528443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8435.540153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8436.765698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8438.450827] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8443.756196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8443.911573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8445.389262] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8447.532113] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8448.910498] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8452.808758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8452.937808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8455.046399] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8457.086878] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8462.184737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8467.427758] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8467.428706] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8467.435987] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8467.438873] Lustre: Skipped 1 previous similar message [ 8470.157356] LustreError: 749905:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8470.186313] Lustre: server umount lustre-MDT0001 complete [ 8472.546034] LustreError: 748144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8472.547229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8472.547521] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8472.547533] Lustre: Skipped 1 previous similar message [ 8472.548858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8472.548866] Lustre: Skipped 1 previous similar message [ 8472.555123] LustreError: 748144:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8477.665643] LustreError: 748144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8477.666605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8478.068280] LustreError: 750107:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8478.073637] LustreError: 750107:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8478.124290] Lustre: server umount lustre-OST0000 complete [ 8479.913038] LustreError: 750356:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8479.914642] LustreError: 750356:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8479.966588] Lustre: server umount lustre-MDT0000 complete [ 8485.704352] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8486.650609] Key type lgssc unregistered [ 8486.771389] LNet: 751232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8486.773251] LNetError: 751232:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8486.779547] LNet: Removed LNI 192.168.204.123@tcp [ 8487.069914] Key type .llcrypt unregistered [ 8487.070796] Key type ._llcrypt unregistered [ 8505.173946] Key type ._llcrypt registered [ 8505.175539] Key type .llcrypt registered [ 8505.226675] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 8533.639956] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8534.099260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8534.107734] alg: No test for adler32 (adler32-zlib) [ 8534.953365] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 8535.031576] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 8536.608159] Key type lgssc registered [ 8537.059536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8542.825694] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8543.073413] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8543.673542] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8544.533805] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8567.270351] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8573.489232] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8578.163141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8578.179264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8578.185308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8579.279150] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8579.291830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8579.330162] Lustre: lustre-MDT0000: new disk, initializing [ 8579.357076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8579.365073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8580.715783] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8584.528671] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8589.425767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8589.449580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8589.544950] Lustre: lustre-OST0000: new disk, initializing [ 8589.548426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8589.550260] Lustre: Skipped 1 previous similar message [ 8589.575945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8591.477438] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8594.594387] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8595.451641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8595.454359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8595.461820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8603.812102] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8609.797431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8609.832463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8609.891649] Lustre: 764101:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8609.936375] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8609.987423] Lustre: lustre-MDT0001: new disk, initializing [ 8610.030468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8610.044431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8610.047333] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8611.599411] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8615.110866] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8615.170665] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8616.305745] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8620.427773] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 4 sec [ 8624.610233] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8624.610735] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8624.612596] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8624.614931] Lustre: Skipped 1 previous similar message [ 8624.617275] Lustre: Skipped 1 previous similar message [ 8625.871483] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.23@tcp (stopping) [ 8629.730287] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8629.731809] Lustre: Skipped 1 previous similar message [ 8634.853542] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8634.857304] Lustre: Skipped 2 previous similar messages [ 8636.384086] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8636.419293] LustreError: 768435:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8636.441522] Lustre: server umount lustre-MDT0001 complete [ 8645.090364] LustreError: 764108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8645.094223] LustreError: 764108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8650.209715] LustreError: 764113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8650.210219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8650.218860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8650.221525] Lustre: Skipped 1 previous similar message [ 8654.573419] LustreError: 768707:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 8654.576956] LustreError: 768707:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8654.612740] Lustre: server umount lustre-OST0000 complete [ 8655.329968] LustreError: 764113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8662.140572] LustreError: 768956:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 8662.142068] LustreError: 768956:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8662.195261] Lustre: server umount lustre-MDT0000 complete [ 8667.075780] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8667.911498] Key type lgssc unregistered [ 8668.031354] LNet: 769833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8668.033336] LNetError: 769833:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8668.042260] LNet: Removed LNI 192.168.204.123@tcp [ 8668.287238] Key type .llcrypt unregistered [ 8668.288163] Key type ._llcrypt unregistered [ 8684.924510] Key type ._llcrypt registered [ 8684.925627] Key type .llcrypt registered [ 8684.958509] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8685.040265] Key type .llcrypt unregistered [ 8685.041310] Key type ._llcrypt unregistered [ 8707.198644] Key type ._llcrypt registered [ 8707.199622] Key type .llcrypt registered [ 8707.237212] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8707.736743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8707.749461] alg: No test for adler32 (adler32-zlib) [ 8708.612649] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 8708.704575] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 8710.304157] Key type lgssc registered [ 8710.782085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8718.430469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8723.100660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8726.584374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8731.533107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8731.574568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8731.583192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8732.713675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8732.727397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8732.770870] Lustre: lustre-MDT0000: new disk, initializing [ 8732.809508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8732.821255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8734.623758] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8741.605074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8741.644253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8741.685465] Lustre: 778051:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8741.723779] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8741.727617] Lustre: Skipped 1 previous similar message [ 8741.778347] Lustre: lustre-MDT0001: new disk, initializing [ 8741.819990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8741.840623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8741.846425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8743.480780] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8747.175644] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8748.645259] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8753.634803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8753.676608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8753.815278] Lustre: lustre-OST0000: new disk, initializing [ 8753.817744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8753.854605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8756.207897] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8757.329224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8757.335433] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8757.393088] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8760.300277] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8767.457503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8767.460116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8767.464648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8767.972816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8767.976878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8767.978173] Lustre: Skipped 1 previous similar message [ 8767.983748] Lustre: Skipped 1 previous similar message [ 8769.806703] LustreError: 780923:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8769.844686] Lustre: server umount lustre-MDT0000 complete [ 8771.726804] LustreError: 779021:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767332191 with bad export cookie 18004338545782124502 [ 8771.730066] LustreError: 779021:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8771.730782] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8771.816135] LustreError: 781174:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8771.818812] LustreError: 781174:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8771.876483] Lustre: server umount lustre-MDT0001 complete [ 8783.550407] LustreError: 781375:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8783.554400] LustreError: 781375:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8783.632391] Lustre: server umount lustre-OST0000 complete [ 8788.621642] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:36:48 (1767332208) [ 8789.564965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8795.842306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8796.107734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8797.746359] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8801.251082] LustreError: 782588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8801.264231] LustreError: 782588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8803.514990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8803.670146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8805.455563] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8808.025271] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8809.698932] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8814.529731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8814.564389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8814.675911] Lustre: Found index 0 for lustre-OST0000, updating log [ 8814.783574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8817.226460] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8819.618366] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8820.965900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8828.453599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8828.454969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8832.497098] LustreError: 785230:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8832.499758] LustreError: 785230:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8832.548995] Lustre: server umount lustre-OST0000 complete [ 8833.869876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8835.207727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8835.209685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8835.217927] Lustre: Skipped 1 previous similar message [ 8835.220935] Lustre: Skipped 3 previous similar messages [ 8841.358160] LustreError: 785533:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8841.362440] LustreError: 785533:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8841.399907] Lustre: server umount lustre-MDT0000 complete [ 8841.697764] LustreError: 782588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8841.707518] LustreError: 782588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8843.231700] LustreError: 783500:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767332262 with bad export cookie 18004338545782125888 [ 8843.236314] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8843.236488] LustreError: 783500:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8843.368257] Lustre: server umount lustre-MDT0001 complete [ 8847.141459] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 00:37:46 (1767332266) [ 8848.052153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8848.551105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8849.033170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8849.496177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8849.972986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8850.462279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8850.944633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8851.413689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8851.888476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8852.374335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8852.846266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8853.348078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8853.825572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8854.314387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8854.795205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8855.272910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8855.756994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8856.228319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8856.716312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8857.204430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8857.667865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8858.157077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8858.641390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8859.111252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8859.589172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8860.058792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8860.536990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8861.000965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8861.466164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8861.936541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8862.409588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8862.895795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8863.387947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8863.868736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8864.374448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8864.865028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8865.338755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8865.813099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8866.278510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8866.714414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8867.166715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8867.653451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8868.134268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8868.571063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8868.862882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8869.287113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8869.792390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8870.289125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8870.764638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8871.245329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8871.757907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8878.184560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8878.394627] LustreError: 789553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8878.426294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8880.026068] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8882.784333] LNet: 1 local NIs in recovery (showing 1): 192.168.204.123@tcp [ 8883.682532] LustreError: 789553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8885.560116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8887.071922] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8889.498551] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8891.142496] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8896.132653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8896.185152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8896.289480] Lustre: Found index 0 for lustre-OST0000, updating log [ 8899.065851] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8901.618361] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8902.540134] LustreError: 792241:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8902.541778] LustreError: 792241:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8902.581236] Lustre: server umount lustre-OST0000 complete [ 8906.209406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8906.213984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8906.221928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8906.225460] Lustre: Skipped 3 previous similar messages [ 8910.864384] Lustre: server umount lustre-MDT0000 complete [ 8912.192118] LustreError: 790465:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767332331 with bad export cookie 18004338545782126966 [ 8912.195230] LustreError: 790465:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8912.195843] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8912.298661] Lustre: server umount lustre-MDT0001 complete [ 8913.137067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8929.521438] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 8946.272196] LNet: 1 local NIs in recovery (showing 1): 192.168.204.123@tcp [ 8953.159770] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8959.084496] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8959.191556] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8959.575276] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8960.480067] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8962.521537] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 00:39:42 (1767332382) [ 8968.838144] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8969.742647] Key type lgssc unregistered [ 8969.892666] LNet: 801760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8969.898417] LNetError: 801760:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9082.537064] LNet: Removed LNI 192.168.204.123@tcp [ 9083.033892] Key type .llcrypt unregistered [ 9083.035049] Key type ._llcrypt unregistered [ 9112.903249] Key type ._llcrypt registered [ 9112.904775] Key type .llcrypt registered [ 9112.957273] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 9113.320213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9113.354868] alg: No test for adler32 (adler32-zlib) [ 9114.237591] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 9114.334283] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 9115.912187] Key type lgssc registered [ 9116.518544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9118.666273] LNet: Added LNI 192.168.204.123@tcp100 [8/256/0/180] [ 9119.108326] LNet: Added LNI 192.168.204.123@tcp101 [8/256/0/180] [ 9119.560085] LNet: Added LNI 192.168.204.123@tcp102 [8/256/0/180] [ 9120.015503] LNet: Added LNI 192.168.204.123@tcp103 [8/256/0/180] [ 9120.453930] LNet: Added LNI 192.168.204.123@tcp104 [8/256/0/180] [ 9120.899907] LNet: Added LNI 192.168.204.123@tcp105 [8/256/0/180] [ 9121.322993] LNet: Added LNI 192.168.204.123@tcp106 [8/256/0/180] [ 9121.760058] LNet: Added LNI 192.168.204.123@tcp107 [8/256/0/180] [ 9122.210422] LNet: Added LNI 192.168.204.123@tcp108 [8/256/0/180] [ 9122.654081] LNet: Added LNI 192.168.204.123@tcp109 [8/256/0/180] [ 9123.083617] LNet: Added LNI 192.168.204.123@tcp110 [8/256/0/180] [ 9123.477920] LNet: Added LNI 192.168.204.123@tcp111 [8/256/0/180] [ 9123.892812] LNet: Added LNI 192.168.204.123@tcp112 [8/256/0/180] [ 9124.320838] LNet: Added LNI 192.168.204.123@tcp113 [8/256/0/180] [ 9124.735201] LNet: Added LNI 192.168.204.123@tcp114 [8/256/0/180] [ 9125.168864] LNet: Added LNI 192.168.204.123@tcp115 [8/256/0/180] [ 9125.605742] LNet: Added LNI 192.168.204.123@tcp116 [8/256/0/180] [ 9126.048035] LNet: Added LNI 192.168.204.123@tcp117 [8/256/0/180] [ 9126.486647] LNet: Added LNI 192.168.204.123@tcp118 [8/256/0/180] [ 9126.913823] LNet: Added LNI 192.168.204.123@tcp119 [8/256/0/180] [ 9127.352660] LNet: Added LNI 192.168.204.123@tcp120 [8/256/0/180] [ 9127.775924] LNet: Added LNI 192.168.204.123@tcp121 [8/256/0/180] [ 9128.208470] LNet: Added LNI 192.168.204.123@tcp122 [8/256/0/180] [ 9128.651264] LNet: Added LNI 192.168.204.123@tcp123 [8/256/0/180] [ 9129.081362] LNet: Added LNI 192.168.204.123@tcp124 [8/256/0/180] [ 9129.506733] LNet: Added LNI 192.168.204.123@tcp125 [8/256/0/180] [ 9129.947316] LNet: Added LNI 192.168.204.123@tcp126 [8/256/0/180] [ 9130.387205] LNet: Added LNI 192.168.204.123@tcp127 [8/256/0/180] [ 9130.724390] LNet: Added LNI 192.168.204.123@tcp128 [8/256/0/180] [ 9130.982252] LNet: Added LNI 192.168.204.123@tcp129 [8/256/0/180] [ 9131.317333] LNet: Added LNI 192.168.204.123@tcp130 [8/256/0/180] [ 9131.746483] LNet: Added LNI 192.168.204.123@tcp131 [8/256/0/180] [ 9132.184598] LNet: Added LNI 192.168.204.123@tcp132 [8/256/0/180] [ 9132.614488] LNet: Added LNI 192.168.204.123@tcp133 [8/256/0/180] [ 9133.053261] LNet: Added LNI 192.168.204.123@tcp134 [8/256/0/180] [ 9133.490802] LNet: Added LNI 192.168.204.123@tcp135 [8/256/0/180] [ 9146.654782] Lustre: DEBUG MARKER: conf-sanity test_73c: @@@@@@ FAIL: unable to start ost1 [ 9158.247406] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 9159.712558] Key type lgssc unregistered [ 9159.926039] LNet: 810028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9159.931427] LNetError: 810028:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9159.935679] LNet: Removed LNI 192.168.204.123@tcp [ 9159.937911] LNet: Removed LNI 192.168.204.123@tcp100 [ 9159.940404] LNet: Removed LNI 192.168.204.123@tcp101 [ 9159.942694] LNet: Removed LNI 192.168.204.123@tcp102 [ 9159.945190] LNet: Removed LNI 192.168.204.123@tcp103 [ 9159.947550] LNet: Removed LNI 192.168.204.123@tcp104 [ 9159.949899] LNet: Removed LNI 192.168.204.123@tcp105 [ 9159.952351] LNet: Removed LNI 192.168.204.123@tcp106 [ 9159.954676] LNet: Removed LNI 192.168.204.123@tcp107 [ 9159.957074] LNet: Removed LNI 192.168.204.123@tcp108 [ 9159.959412] LNet: Removed LNI 192.168.204.123@tcp109 [ 9159.961748] LNet: Removed LNI 192.168.204.123@tcp110 [ 9159.964200] LNet: Removed LNI 192.168.204.123@tcp111 [ 9159.966591] LNet: Removed LNI 192.168.204.123@tcp112 [ 9159.968922] LNet: Removed LNI 192.168.204.123@tcp113 [ 9159.971795] LNet: Removed LNI 192.168.204.123@tcp114 [ 9159.974544] LNet: Removed LNI 192.168.204.123@tcp115 [ 9159.976609] LNet: Removed LNI 192.168.204.123@tcp116 [ 9159.978756] LNet: Removed LNI 192.168.204.123@tcp117 [ 9159.980904] LNet: Removed LNI 192.168.204.123@tcp118 [ 9159.983199] LNet: Removed LNI 192.168.204.123@tcp119 [ 9159.985234] LNet: Removed LNI 192.168.204.123@tcp120 [ 9159.987188] LNet: Removed LNI 192.168.204.123@tcp121 [ 9159.989147] LNet: Removed LNI 192.168.204.123@tcp122 [ 9159.991153] LNet: Removed LNI 192.168.204.123@tcp123 [ 9159.993124] LNet: Removed LNI 192.168.204.123@tcp124 [ 9159.994818] LNet: Removed LNI 192.168.204.123@tcp125 [ 9159.996622] LNet: Removed LNI 192.168.204.123@tcp126 [ 9159.998558] LNet: Removed LNI 192.168.204.123@tcp127 [ 9160.000344] LNet: Removed LNI 192.168.204.123@tcp128 [ 9160.002168] LNet: Removed LNI 192.168.204.123@tcp129 [ 9160.003843] LNet: Removed LNI 192.168.204.123@tcp130 [ 9160.005521] LNet: Removed LNI 192.168.204.123@tcp131 [ 9160.007188] LNet: Removed LNI 192.168.204.123@tcp132 [ 9160.008782] LNet: Removed LNI 192.168.204.123@tcp133 [ 9160.010405] LNet: Removed LNI 192.168.204.123@tcp134 [ 9160.017438] LNet: Removed LNI 192.168.204.123@tcp135 [ 9160.413814] Key type .llcrypt unregistered [ 9160.416487] Key type ._llcrypt unregistered [ 9175.745420] Key type ._llcrypt registered [ 9175.747278] Key type .llcrypt registered [ 9175.809592] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 9209.869372] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 9210.236785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9210.279945] alg: No test for adler32 (adler32-zlib) [ 9211.142484] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 9211.254497] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 9212.856183] Key type lgssc registered [ 9213.477884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9220.601087] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9221.075191] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9221.940100] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9223.058449] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9226.431282] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 00:44:05 (1767332645) [ 9237.085380] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 9238.704522] Key type lgssc unregistered [ 9238.871799] LNet: 818934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9238.876383] LNetError: 818934:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9238.888677] LNet: Removed LNI 192.168.204.123@tcp [ 9239.353254] Key type .llcrypt unregistered [ 9239.355300] Key type ._llcrypt unregistered [ 9273.373763] Key type ._llcrypt registered [ 9273.375526] Key type .llcrypt registered [ 9273.444506] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 9273.753385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9273.790093] alg: No test for adler32 (adler32-zlib) [ 9274.685935] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 9274.797448] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 9276.416202] Key type lgssc registered [ 9277.004849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9293.097571] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing lnet_if_list [ 9295.419919] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 9295.436297] LNet: Added LNI 192.168.204.123@tcp10 [8/256/0/180] [ 9297.981424] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 9298.000333] LNet: Added LNI 192.168.204.123@tcp11 [8/256/0/180] [ 9299.689694] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 9299.708511] LNet: Added LNI 192.168.204.123@tcp12 [8/256/0/180] [ 9301.448314] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 9301.460830] LNet: Added LNI 192.168.204.123@tcp13 [8/256/0/180] [ 9303.128573] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 9303.150383] LNet: Added LNI 192.168.204.123@tcp14 [8/256/0/180] [ 9305.076371] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 9305.094409] LNet: Added LNI 192.168.204.123@tcp15 [8/256/0/180] [ 9307.120755] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 9307.136670] LNet: Added LNI 192.168.204.123@tcp16 [8/256/0/180] [ 9309.174974] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 9309.195780] LNet: Added LNI 192.168.204.123@tcp17 [8/256/0/180] [ 9311.328684] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 9311.346830] LNet: Added LNI 192.168.204.123@tcp18 [8/256/0/180] [ 9313.491205] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 9313.510853] LNet: Added LNI 192.168.204.123@tcp19 [8/256/0/180] [ 9315.696458] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 9315.710701] LNet: Added LNI 192.168.204.123@tcp20 [8/256/0/180] [ 9317.898500] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 9317.917375] LNet: Added LNI 192.168.204.123@tcp21 [8/256/0/180] [ 9320.253931] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 9320.272675] LNet: Added LNI 192.168.204.123@tcp22 [8/256/0/180] [ 9322.581735] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 9322.591561] LNet: Added LNI 192.168.204.123@tcp23 [8/256/0/180] [ 9324.709924] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 9324.719513] LNet: Added LNI 192.168.204.123@tcp24 [8/256/0/180] [ 9327.321340] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 9327.340710] LNet: Added LNI 192.168.204.123@tcp25 [8/256/0/180] [ 9330.106759] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 9330.126359] LNet: Added LNI 192.168.204.123@tcp26 [8/256/0/180] [ 9332.921364] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 9332.940833] LNet: Added LNI 192.168.204.123@tcp27 [8/256/0/180] [ 9335.659184] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 9335.678777] LNet: Added LNI 192.168.204.123@tcp28 [8/256/0/180] [ 9338.270117] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 9338.282182] LNet: Added LNI 192.168.204.123@tcp29 [8/256/0/180] [ 9341.101374] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 9341.120756] LNet: Added LNI 192.168.204.123@tcp30 [8/256/0/180] [ 9344.222957] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 9344.241961] LNet: Added LNI 192.168.204.123@tcp31 [8/256/0/180] [ 9347.440995] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 9347.459829] LNet: Added LNI 192.168.204.123@tcp32 [8/256/0/180] [ 9350.559885] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 9350.579732] LNet: Added LNI 192.168.204.123@tcp33 [8/256/0/180] [ 9353.451153] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 9353.470696] LNet: Added LNI 192.168.204.123@tcp34 [8/256/0/180] [ 9356.566137] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 9356.586948] LNet: Added LNI 192.168.204.123@tcp35 [8/256/0/180] [ 9359.992715] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 9360.013699] LNet: Added LNI 192.168.204.123@tcp36 [8/256/0/180] [ 9363.573837] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 9363.595277] LNet: Added LNI 192.168.204.123@tcp37 [8/256/0/180] [ 9367.121344] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 9367.140627] LNet: Added LNI 192.168.204.123@tcp38 [8/256/0/180] [ 9370.591620] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 9370.612136] LNet: Added LNI 192.168.204.123@tcp39 [8/256/0/180] [ 9374.169341] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 9374.186859] LNet: Added LNI 192.168.204.123@tcp40 [8/256/0/180] [ 9377.838793] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 9377.857136] LNet: Added LNI 192.168.204.123@tcp41 [8/256/0/180] [ 9381.310234] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 9381.329497] LNet: Added LNI 192.168.204.123@tcp42 [8/256/0/180] [ 9384.922368] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 9384.937850] LNet: Added LNI 192.168.204.123@tcp43 [8/256/0/180] [ 9388.823758] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 9388.842827] LNet: Added LNI 192.168.204.123@tcp44 [8/256/0/180] [ 9392.634587] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 9392.653870] LNet: Added LNI 192.168.204.123@tcp45 [8/256/0/180] [ 9396.349383] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 9396.368910] LNet: Added LNI 192.168.204.123@tcp46 [8/256/0/180] [ 9400.530243] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 9400.549049] LNet: Added LNI 192.168.204.123@tcp47 [8/256/0/180] [ 9404.631075] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 9404.649634] LNet: Added LNI 192.168.204.123@tcp48 [8/256/0/180] [ 9408.836704] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 9408.856553] LNet: Added LNI 192.168.204.123@tcp49 [8/256/0/180] [ 9413.079343] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 9413.098178] LNet: Added LNI 192.168.204.123@tcp50 [8/256/0/180] [ 9580.613283] Lustre: DEBUG MARKER: conf-sanity test_73e: @@@@@@ FAIL: Mount fails on 192.168.204.123@tcp50 [ 9592.541476] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 9594.028253] Key type lgssc unregistered [ 9594.228413] LNet: 846966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9594.231794] LNetError: 846966:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9594.234492] LNet: Removed LNI 192.168.204.123@tcp [ 9594.235532] LNet: Removed LNI 192.168.204.123@tcp10 [ 9594.236698] LNet: Removed LNI 192.168.204.123@tcp11 [ 9594.237782] LNet: Removed LNI 192.168.204.123@tcp12 [ 9594.238814] LNet: Removed LNI 192.168.204.123@tcp13 [ 9594.239969] LNet: Removed LNI 192.168.204.123@tcp14 [ 9594.240957] LNet: Removed LNI 192.168.204.123@tcp15 [ 9594.242733] LNet: Removed LNI 192.168.204.123@tcp16 [ 9594.244335] LNet: Removed LNI 192.168.204.123@tcp17 [ 9594.245475] LNet: Removed LNI 192.168.204.123@tcp18 [ 9594.246533] LNet: Removed LNI 192.168.204.123@tcp19 [ 9594.247529] LNet: Removed LNI 192.168.204.123@tcp20 [ 9594.249352] LNet: Removed LNI 192.168.204.123@tcp21 [ 9594.251091] LNet: Removed LNI 192.168.204.123@tcp22 [ 9594.252108] LNet: Removed LNI 192.168.204.123@tcp23 [ 9594.253124] LNet: Removed LNI 192.168.204.123@tcp24 [ 9594.254417] LNet: Removed LNI 192.168.204.123@tcp25 [ 9594.255549] LNet: Removed LNI 192.168.204.123@tcp26 [ 9594.256801] LNet: Removed LNI 192.168.204.123@tcp27 [ 9594.258051] LNet: Removed LNI 192.168.204.123@tcp28 [ 9594.259077] LNet: Removed LNI 192.168.204.123@tcp29 [ 9594.260127] LNet: Removed LNI 192.168.204.123@tcp30 [ 9594.261250] LNet: Removed LNI 192.168.204.123@tcp31 [ 9594.262392] LNet: Removed LNI 192.168.204.123@tcp32 [ 9594.263544] LNet: Removed LNI 192.168.204.123@tcp33 [ 9594.264706] LNet: Removed LNI 192.168.204.123@tcp34 [ 9594.265802] LNet: Removed LNI 192.168.204.123@tcp35 [ 9594.266776] LNet: Removed LNI 192.168.204.123@tcp36 [ 9594.267914] LNet: Removed LNI 192.168.204.123@tcp37 [ 9594.268894] LNet: Removed LNI 192.168.204.123@tcp38 [ 9594.269961] LNet: Removed LNI 192.168.204.123@tcp39 [ 9594.271070] LNet: Removed LNI 192.168.204.123@tcp40 [ 9594.272069] LNet: Removed LNI 192.168.204.123@tcp41 [ 9594.273069] LNet: Removed LNI 192.168.204.123@tcp42 [ 9594.274169] LNet: Removed LNI 192.168.204.123@tcp43 [ 9594.275294] LNet: Removed LNI 192.168.204.123@tcp44 [ 9594.276410] LNet: Removed LNI 192.168.204.123@tcp45 [ 9594.277389] LNet: Removed LNI 192.168.204.123@tcp46 [ 9594.278365] LNet: Removed LNI 192.168.204.123@tcp47 [ 9594.279340] LNet: Removed LNI 192.168.204.123@tcp48 [ 9594.280344] LNet: Removed LNI 192.168.204.123@tcp49 [ 9594.287292] LNet: Removed LNI 192.168.204.123@tcp50 [ 9594.643571] Key type .llcrypt unregistered [ 9594.645614] Key type ._llcrypt unregistered [ 9631.223497] Key type ._llcrypt registered [ 9631.225381] Key type .llcrypt registered [ 9631.276197] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 9631.644381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9631.672198] alg: No test for adler32 (adler32-zlib) [ 9632.558366] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 9632.664594] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 9634.248202] Key type lgssc registered [ 9634.855630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9639.872181] Lustre: DEBUG MARKER: conf-sanity test_74: @@@@@@ FAIL: MDT start failed [ 9642.522779] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 00:51:01 (1767333061) [ 9676.136565] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 9682.013760] Lustre: DEBUG MARKER: conf-sanity test_75: @@@@@@ FAIL: add mds1 failed for default params [ 9697.436222] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 9728.406042] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 9736.031555] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9736.422354] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9737.272737] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9738.217301] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9739.097276] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9421 sec ========== 00:52:38 (1767333158) [ 9740.005878] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:52:39 (1767333159) === [ 9741.709517] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:52:41 (1767333161) === [ 9756.268214] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 9757.892568] Key type lgssc unregistered [ 9758.060411] LNet: 864517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9758.062383] LNetError: 864517:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9758.070554] LNet: Removed LNI 192.168.204.123@tcp [ 9758.484641] Key type .llcrypt unregistered [ 9758.486581] Key type ._llcrypt unregistered