[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 314935184 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.519084] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.521448] pid_max: default: 32768 minimum: 301 [ 0.522624] Security Framework initialized [ 0.523654] SELinux: Initializing. [ 0.524988] SELinux: Starting in permissive mode [ 0.526546] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.530101] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.532385] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.534564] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.537298] Initializing cgroup subsys memory [ 0.538305] Initializing cgroup subsys devices [ 0.539426] Initializing cgroup subsys freezer [ 0.540488] Initializing cgroup subsys net_cls [ 0.541496] Initializing cgroup subsys blkio [ 0.542416] Initializing cgroup subsys perf_event [ 0.543430] Initializing cgroup subsys hugetlb [ 0.544412] Initializing cgroup subsys pids [ 0.545784] Initializing cgroup subsys net_prio [ 0.547177] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.549825] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.552414] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.554084] tlb_flushall_shift: 6 [ 0.555072] FEATURE SPEC_CTRL Present [ 0.556289] FEATURE IBPB_SUPPORT Present [ 0.557219] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.558552] Spectre V2 : Vulnerable [ 0.559279] Speculative Store Bypass: Vulnerable [ 0.561109] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.567793] ACPI: Core revision 20130517 [ 0.570251] ACPI: All ACPI Tables successfully acquired [ 0.571745] ftrace: allocating 30294 entries in 119 pages [ 0.637830] Enabling x2apic [ 0.638642] Enabled x2apic [ 0.639866] Switched APIC routing to physical x2apic. [ 0.643494] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.646471] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.651280] TSC deadline timer enabled [ 0.651528] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.654428] ... version: 2 [ 0.655353] ... bit width: 48 [ 0.656293] ... generic registers: 4 [ 0.657117] ... value mask: 0000ffffffffffff [ 0.658968] ... max period: 00007fffffffffff [ 0.660156] ... fixed-purpose events: 3 [ 0.661374] ... event mask: 000000070000000f [ 0.662953] KVM setup paravirtual spinlock [ 0.668051] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.671995] KVM setup async PF for cpu 1 [ 0.673457] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.676279] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.679684] KVM setup async PF for cpu 2 [ 0.666549] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.680219] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.684289] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.686380] Brought up 4 CPUs [ 0.686413] KVM setup async PF for cpu 3 [ 0.686419] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.690949] smpboot: Max logical packages: 1 [ 0.692486] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.697803] devtmpfs: initialized [ 0.699488] x86/mm: Memory block size: 128MB [ 0.705558] EVM: security.selinux [ 0.706779] EVM: security.ima [ 0.707745] EVM: security.capability [ 0.711482] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.714266] NET: Registered protocol family 16 [ 0.716218] cpuidle: using governor haltpoll [ 0.718339] ACPI: bus type PCI registered [ 0.719792] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.722280] PCI: Using configuration type 1 for base access [ 0.724295] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.733771] ACPI: Added _OSI(Module Device) [ 0.735124] ACPI: Added _OSI(Processor Device) [ 0.736506] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.738530] ACPI: Added _OSI(Processor Aggregator Device) [ 0.739857] ACPI: Added _OSI(Linux-Dell-Video) [ 0.742725] ACPI: EC: Look up EC in DSDT [ 0.745506] ACPI: Interpreter enabled [ 0.746525] ACPI: (supports S0 S3 S4 S5) [ 0.747494] ACPI: Using IOAPIC for interrupt routing [ 0.749331] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.752595] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.759910] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.761981] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.763599] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.764982] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.771494] acpiphp: Slot [2] registered [ 0.773432] acpiphp: Slot [3] registered [ 0.775134] acpiphp: Slot [4] registered [ 0.776922] acpiphp: Slot [5] registered [ 0.777882] acpiphp: Slot [6] registered [ 0.779109] acpiphp: Slot [7] registered [ 0.780005] acpiphp: Slot [8] registered [ 0.780937] acpiphp: Slot [9] registered [ 0.781879] acpiphp: Slot [10] registered [ 0.782760] acpiphp: Slot [11] registered [ 0.783651] acpiphp: Slot [12] registered [ 0.784589] acpiphp: Slot [13] registered [ 0.785541] acpiphp: Slot [14] registered [ 0.786394] acpiphp: Slot [15] registered [ 0.787436] acpiphp: Slot [16] registered [ 0.788327] acpiphp: Slot [17] registered [ 0.789963] acpiphp: Slot [18] registered [ 0.791178] acpiphp: Slot [19] registered [ 0.792972] acpiphp: Slot [20] registered [ 0.794422] acpiphp: Slot [21] registered [ 0.796180] acpiphp: Slot [22] registered [ 0.797327] acpiphp: Slot [23] registered [ 0.798946] acpiphp: Slot [24] registered [ 0.800198] acpiphp: Slot [25] registered [ 0.801415] acpiphp: Slot [26] registered [ 0.802801] acpiphp: Slot [27] registered [ 0.804220] acpiphp: Slot [28] registered [ 0.805676] acpiphp: Slot [29] registered [ 0.807256] acpiphp: Slot [30] registered [ 0.808631] acpiphp: Slot [31] registered [ 0.810140] PCI host bridge to bus 0000:00 [ 0.811571] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.814103] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.816649] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.819170] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.821824] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.823592] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.825419] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.826558] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.827823] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.836707] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.838756] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.842657] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.845809] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.850376] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.852815] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.853996] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.857529] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.861835] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.864073] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.876474] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.880584] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.883281] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.886971] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.890319] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.900175] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.906764] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.909832] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.912356] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.926671] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.931023] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.934091] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.936726] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.945043] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.948377] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.951593] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.955968] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.965573] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.969114] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.971859] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.974353] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.983199] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.989758] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.992537] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.994990] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.006186] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 1.011319] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 1.014700] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.017077] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.019447] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 1.021780] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.025338] vgaarb: loaded [ 1.026550] SCSI subsystem initialized [ 1.027982] ACPI: bus type USB registered [ 1.029443] usbcore: registered new interface driver usbfs [ 1.031380] usbcore: registered new interface driver hub [ 1.033308] usbcore: registered new device driver usb [ 1.035352] PCI: Using ACPI for IRQ routing [ 1.036826] PCI: pci_cache_line_size set to 64 bytes [ 1.037146] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.037152] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 1.037154] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 1.037482] NetLabel: Initializing [ 1.038809] NetLabel: domain hash size = 128 [ 1.040386] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.042197] NetLabel: unlabeled traffic allowed by default [ 1.044477] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.046342] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.051650] amd_nb: Cannot enumerate AMD northbridges [ 1.053514] Switched to clocksource kvm-clock [ 1.074451] pnp: PnP ACPI init [ 1.075743] ACPI: bus type PNP registered [ 1.077332] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.077421] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.077466] pnp 00:02: [dma 2] [ 1.077524] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.077611] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.077698] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.077779] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.078199] pnp: PnP ACPI: found 6 devices [ 1.079668] ACPI: bus type PNP unregistered [ 1.090567] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.090573] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.090576] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.090580] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.090583] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 1.090648] NET: Registered protocol family 2 [ 1.092758] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.095728] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.099068] TCP: Hash tables configured (established 32768 bind 32768) [ 1.101453] TCP: reno registered [ 1.102255] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.103730] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.105358] NET: Registered protocol family 1 [ 1.106746] RPC: Registered named UNIX socket transport module. [ 1.108057] RPC: Registered udp transport module. [ 1.109284] RPC: Registered tcp transport module. [ 1.110329] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.111703] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.113024] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.114960] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.117034] PCI: CLS 0 bytes, default 64 [ 1.117147] Unpacking initramfs... [ 2.451236] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 2.454279] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.455661] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 2.458285] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 2.460173] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 2.461341] RAPL PMU: hw unit of domain package 2^-0 Joules [ 2.462544] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 2.465587] cryptomgr_test (50) used greatest stack depth: 14128 bytes left [ 2.466045] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.466092] Initialise system trusted keyring [ 2.499868] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 2.502440] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.508996] zpool: loaded [ 2.509631] zbud: loaded [ 2.510667] VFS: Disk quotas dquot_6.6.0 [ 2.512465] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.515925] NFS: Registering the id_resolver key type [ 2.517897] Key type id_resolver registered [ 2.519475] Key type id_legacy registered [ 2.520921] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.523990] Key type big_key registered [ 2.525273] SELinux: Registering netfilter hooks [ 2.526714] cryptomgr_test (56) used greatest stack depth: 13968 bytes left [ 2.529092] cryptomgr_test (59) used greatest stack depth: 13664 bytes left [ 2.529867] NET: Registered protocol family 38 [ 2.529881] Key type asymmetric registered [ 2.529886] Asymmetric key parser 'x509' registered [ 2.530109] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.530221] io scheduler noop registered [ 2.530226] io scheduler deadline registered (default) [ 2.530284] io scheduler cfq registered [ 2.530288] io scheduler mq-deadline registered [ 2.530292] io scheduler kyber registered [ 2.533408] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.533424] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.544887] intel_idle: does not run on family 6 model 62 [ 2.545039] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.547304] ACPI: Power Button [PWRF] [ 2.549067] GHES: HEST is not enabled! [ 2.609569] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.671736] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 2.803752] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 2.875367] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 3.015350] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.045675] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 3.074872] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 3.078544] Non-volatile memory driver v1.3 [ 3.080372] Linux agpgart interface v0.103 [ 3.082498] crash memory driver: version 1.1 [ 3.084895] nbd: registered device at major 43 [ 3.090283] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 3.090342] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 3.090391] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 3.090442] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 3.090526] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 3.097816] virtio_blk virtio1: [vda] 116096 512-byte logical blocks (59.4 MB/56.6 MiB) [ 3.105608] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 3.105675] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 3.105715] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 3.105754] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 3.105790] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 3.112903] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.118114] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 3.118145] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 3.118179] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 3.118205] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 3.118230] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 3.124185] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.129143] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 3.129199] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 3.129257] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 3.129296] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 3.129323] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 3.135644] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.143914] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 3.143953] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 3.143982] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 3.144033] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 3.144062] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 3.150556] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.158624] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 3.158694] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 3.158756] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 3.158800] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 3.158850] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 3.166032] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.171831] rdac: device handler registered [ 3.174168] hp_sw: device handler registered [ 3.175650] emc: device handler registered [ 3.177115] libphy: Fixed MDIO Bus: probed [ 3.181253] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 3.183403] ehci-pci: EHCI PCI platform driver [ 3.184654] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 3.186140] ohci-pci: OHCI PCI platform driver [ 3.187621] uhci_hcd: USB Universal Host Controller Interface driver [ 3.189616] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.192601] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.193911] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.195474] mousedev: PS/2 mouse device common for all mice [ 3.199497] rtc_cmos 00:05: RTC can wake from S4 [ 3.200206] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.207845] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 3.210442] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.213883] hidraw: raw HID events driver (C) Jiri Kosina [ 3.216346] usbcore: registered new interface driver usbhid [ 3.218221] usbhid: USB HID core driver [ 3.219784] drop_monitor: Initializing network drop monitor service [ 3.221931] Netfilter messages via NETLINK v0.30. [ 3.223902] TCP: cubic registered [ 3.225236] Initializing XFRM netlink socket [ 3.227553] NET: Registered protocol family 10 [ 3.229931] NET: Registered protocol family 17 [ 3.231299] Key type dns_resolver registered [ 3.232948] mce: Using 10 MCE banks [ 3.234376] Loading compiled-in X.509 certificates [ 3.237093] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 3.239723] registered taskstats version 1 [ 3.243912] modprobe (71) used greatest stack depth: 13456 bytes left [ 3.249457] Key type trusted registered [ 3.255683] Key type encrypted registered [ 3.257299] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 3.262026] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 3.265886] rtc_cmos 00:05: setting system clock to 2025-06-11 01:29:57 UTC (1749605397) [ 3.268865] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 3.270570] Write protecting the kernel read-only data: 12288k [ 3.272839] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 3.274972] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 3.284829] random: systemd: uninitialized urandom read (16 bytes read) [ 3.289242] random: systemd: uninitialized urandom read (16 bytes read) [ 3.290948] random: systemd: uninitialized urandom read (16 bytes read) [ 3.295117] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 3.299845] systemd[1]: Detected virtualization kvm. [ 3.301323] systemd[1]: Detected architecture x86-64. [ 3.302551] systemd[1]: Running in initial RAM disk. [ 3.306732] systemd[1]: No hostname configured. [ 3.308311] systemd[1]: Set hostname to . [ 3.309852] random: systemd: uninitialized urandom read (16 bytes read) [ 3.312021] systemd[1]: Initializing machine ID from random generator. [ 3.326941] mkdir (89) used greatest stack depth: 13392 bytes left [ 3.360402] dracut-rootfs-g (86) used greatest stack depth: 13264 bytes left [ 3.364111] random: systemd: uninitialized urandom read (16 bytes read) [ 3.367573] random: systemd: uninitialized urandom read (16 bytes read) [ 3.369827] random: systemd: uninitialized urandom read (16 bytes read) [ 3.372738] random: systemd: uninitialized urandom read (16 bytes read) [ 3.376734] random: systemd: uninitialized urandom read (16 bytes read) [ 3.378991] random: systemd: uninitialized urandom read (16 bytes read) [ 3.390669] systemd[1]: Created slice Root Slice. [ 3.394894] systemd[1]: Reached target Local File Systems. [ 3.399334] systemd[1]: Reached target Swap. [ 3.403135] systemd[1]: Listening on Journal Socket. [ 3.407523] systemd[1]: Created slice System Slice. [ 3.415930] systemd[1]: Starting Journal Service... [ 3.422173] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 3.432922] systemd[1]: Starting Setup Virtual Console... [ 3.438331] systemd[1]: Reached target Timers. [ 3.443053] systemd[1]: Reached target Slices. [ 3.447808] systemd[1]: Listening on udev Control Socket. [ 3.455395] systemd[1]: Starting dracut cmdline hook... [ 3.461819] systemd[1]: Listening on udev Kernel Socket. [ 3.467060] tsc: Refined TSC clocksource calibration: 2399.986 MHz [ 3.472611] systemd[1]: Reached target Sockets. [ 3.480536] systemd[1]: Starting Load Kernel Modules... [ 3.487495] systemd[1]: Started Journal Service. [ 3.522300] random: fast init done [ 3.948830] libata version 3.00 loaded. [ 3.961900] ata_piix 0000:00:01.1: version 2.13 [ 3.965218] scsi host0: ata_piix [ 3.976957] scsi host1: ata_piix [ 3.991079] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 3.994044] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 4.050811] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 4.191774] ip (320) used greatest stack depth: 13080 bytes left [ 4.241730] ip (343) used greatest stack depth: 12336 bytes left [ 7.257378] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 7.673906] systemd-journald[100]: Received SIGTERM from PID 1 (systemd). [ 7.912888] SELinux: Disabled at runtime. [ 7.913901] SELinux: Unregistering netfilter hooks [ 7.970708] ip_tables: (C) 2000-2006 Netfilter Core Team [ 7.974253] systemd[1]: Inserted module 'ip_tables' [ 8.000256] systemd-cryptse (549) used greatest stack depth: 11680 bytes left [ 8.314389] systemd-journald[563]: Received request to flush runtime journal from PID 1 [ 8.421151] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 8.448645] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 8.498218] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 8.504066] cryptd: max_cpu_qlen set to 1000 [ 8.561232] AVX version of gcm_enc/dec engaged. [ 8.562425] AES CTR mode by8 optimization enabled [ 8.567999] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 8.575936] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 8.578103] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 8.735852] EDAC MC: Ver: 3.0.0 [ 8.751876] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 8.751888] EDAC sbridge: Ver: 1.1.2 [ 11.555691] mount.nfs (766) used greatest stack depth: 10704 bytes left [ 21.780577] device-mapper: uevent: version 1.0.3 [ 21.781953] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 25.750149] libcfs: loading out-of-tree module taints kernel. [ 25.751889] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 25.775908] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 30.468924] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 30.668881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 30.672974] alg: No test for adler32 (adler32-zlib) [ 31.568343] Lustre: Lustre: Build Version: 2.16.56_1_g45307c0 [ 31.777795] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 31.780520] LNet: Accept secure, port 988 [ 33.327684] Key type lgssc registered [ 33.646203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 36.495049] icp: module license 'CDDL' taints kernel. [ 36.497050] Disabling lock debugging due to kernel taint [ 37.935835] ZFS: Loaded module v2.1.5-1, ZFS pool version 5000, ZFS filesystem version 5 [ 40.971399] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 45.555176] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 46.462765] random: crng init done [ 47.566676] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 47.572822] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 49.705839] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 49.709947] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 52.855314] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 56.041089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 56.060979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 56.067781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 57.193672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 57.214228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 57.268126] Lustre: lustre-MDT0000: new disk, initializing [ 57.296556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 57.303318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 57.350866] mount.lustre (7183) used greatest stack depth: 9584 bytes left [ 58.365143] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 62.867830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 62.910147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 62.944494] Lustre: 7201:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 62.958580] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 62.961977] Lustre: Skipped 1 previous similar message [ 63.005878] Lustre: lustre-MDT0001: new disk, initializing [ 63.030034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 63.042687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 63.050458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 64.262775] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 66.695594] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 69.997090] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 70.003215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 70.029482] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 70.035152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 70.127426] Lustre: lustre-OST0000: new disk, initializing [ 70.131811] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 70.151115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 71.906548] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 73.331216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 73.338132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 73.347621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 76.478663] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 76.483370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 76.498738] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 76.501754] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 76.527356] Lustre: lustre-OST0001: new disk, initializing [ 76.531108] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 76.549116] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 77.956249] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 83.163922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 83.167862] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 83.189054] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 83.550224] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 87.933282] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 94.414702] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing check_logdir /tmp/testlogs/ [ 95.258259] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing yml_node [ 96.325254] Lustre: DEBUG MARKER: Client: 2.16.56.1 [ 96.949026] Lustre: DEBUG MARKER: MDS: 2.16.56.1 [ 97.580440] Lustre: DEBUG MARKER: OSS: 2.16.56.1 [ 97.968369] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 10 21:31:31 EDT 2025 [ 102.445661] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 102.806442] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 118.106875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 118.110116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 118.116028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 119.666874] LustreError: 14277:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 119.688761] Lustre: server umount lustre-MDT0000 complete [ 120.716055] LustreError: 7191:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749605514 with bad export cookie 18254736784716616210 [ 120.716820] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 120.722141] LustreError: 7191:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 123.227166] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 123.232132] Lustre: Skipped 4 previous similar messages [ 123.232228] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 123.232230] Lustre: Skipped 3 previous similar messages [ 126.234962] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 126.238425] Lustre: Skipped 2 previous similar messages [ 126.764497] LustreError: 14548:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 126.767831] LustreError: 14548:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 126.839278] Lustre: server umount lustre-MDT0001 complete [ 133.958684] LustreError: 14819:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 133.960529] LustreError: 14819:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 133.968712] Lustre: server umount lustre-OST0000 complete [ 140.897593] LustreError: 15088:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 140.900521] LustreError: 15088:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 140.950178] Lustre: server umount lustre-OST0001 complete [ 142.214176] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 143.895506] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 146.070056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 147.671688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 149.170501] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 149.174436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 150.675195] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 150.678905] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 152.420597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 152.436424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 152.515821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 152.527748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 152.569093] Lustre: lustre-MDT0000: new disk, initializing [ 152.588376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 152.593215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 153.320153] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 156.747303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 156.765760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 156.790808] Lustre: 17745:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 156.875822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 156.899667] mount.lustre (18681) used greatest stack depth: 9472 bytes left [ 157.651323] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 160.185888] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 160.950906] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 162.686158] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 162.690217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 162.706091] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 162.709322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 162.793858] Lustre: lustre-OST0000: new disk, initializing [ 162.795594] Lustre: Skipped 1 previous similar message [ 162.797756] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 162.800035] Lustre: Skipped 2 previous similar messages [ 162.820074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 162.822816] Lustre: Skipped 1 previous similar message [ 163.850867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 163.853522] Lustre: Skipped 1 previous similar message [ 163.855540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 163.871805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 164.060092] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 166.544994] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 167.350689] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 167.401799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 168.159327] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 168.210163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 168.875336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 168.875362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 168.875755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 168.882537] LustreError: Skipped 1 previous similar message [ 173.883421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 173.885160] Lustre: Skipped 2 previous similar messages [ 174.724600] LustreError: 21276:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 174.728974] LustreError: 21276:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 174.790018] Lustre: server umount lustre-OST0000 complete [ 176.891341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 176.893263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 176.896024] Lustre: Skipped 1 previous similar message [ 181.842812] Lustre: server umount lustre-MDT0000 complete [ 182.835613] LustreError: 17736:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749605577 with bad export cookie 18254736784716617729 [ 182.837227] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 182.846302] LustreError: 17736:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 182.892805] LustreError: 21815:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 182.895137] LustreError: 21815:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 182.969803] Lustre: server umount lustre-MDT0001 complete [ 184.199231] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 21:32:57 (1749605577) [ 185.842367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.966375] LustreError: 22499:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 185.972220] LustreError: 22499:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 185.981323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.751338] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 188.843582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 189.630240] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 190.851359] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 191.637115] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 193.337399] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 193.340463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 194.611841] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 195.782738] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 202.344346] Lustre: Modifying parameter general.osc.*.max_dirty_mb=457 in log params [ 208.955143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 208.957182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 208.960151] Lustre: Skipped 1 previous similar message [ 208.961572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 208.963112] Lustre: Skipped 3 previous similar messages [ 211.069562] LustreError: 25337:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 211.071414] LustreError: 25337:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 211.086259] Lustre: server umount lustre-MDT0000 complete [ 211.996290] LustreError: 22481:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749605606 with bad export cookie 18254736784716618611 [ 211.998156] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 212.002372] LustreError: 22481:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 212.114642] Lustre: server umount lustre-MDT0001 complete [ 223.183171] Lustre: server umount lustre-OST0000 complete [ 227.246849] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 229.418332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 229.531861] LustreError: 27471:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 229.542562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.544264] Lustre: Skipped 2 previous similar messages [ 230.244954] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 232.313537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 233.166828] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 233.900435] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 235.509942] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 235.514600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 236.772028] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 238.797752] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 238.802053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 238.815827] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 238.818605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 238.848871] Lustre: lustre-OST0001: new disk, initializing [ 238.850533] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 240.018149] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 240.706485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 240.709076] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 240.729764] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 243.694566] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 250.451234] Lustre: Setting parameter general.obdfilter.*.client_cache_count=256 in log params [ 250.453217] Lustre: Skipped 1 previous similar message [ 252.731603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 252.732120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 252.732122] Lustre: Skipped 2 previous similar messages [ 252.737889] Lustre: Skipped 5 previous similar messages [ 258.775320] LustreError: 31781:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 258.776986] LustreError: 31781:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 258.796003] Lustre: server umount lustre-MDT0000 complete [ 259.714224] LustreError: 27453:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749605653 with bad export cookie 18254736784716619458 [ 259.715858] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 259.719740] LustreError: 27453:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 259.812009] Lustre: server umount lustre-MDT0001 complete [ 270.767918] Lustre: server umount lustre-OST0000 complete [ 281.719750] Lustre: server umount lustre-OST0001 complete [ 284.850838] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 287.049130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 287.155179] LustreError: 1558:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 287.901574] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 289.974230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 290.784691] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 291.508097] Lustre: 2885:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 293.129107] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 293.132668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 294.466475] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 296.596102] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 296.599661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 296.660187] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 296.662098] Lustre: Skipped 6 previous similar messages [ 297.851998] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 304.061226] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 304.880348] Lustre: 5219:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 315.083148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 315.087127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.093635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.095646] Lustre: Skipped 7 previous similar messages [ 317.093672] Lustre: server umount lustre-MDT0000 complete [ 318.078326] LustreError: 1541:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749605712 with bad export cookie 18254736784716620753 [ 318.079257] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 318.087153] LustreError: 1541:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 318.208664] Lustre: server umount lustre-MDT0001 complete [ 329.714573] LustreError: 6029:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 329.716371] LustreError: 6029:0:(obd_class.h:479:obd_check_dev()) Skipped 41 previous similar messages [ 329.725023] Lustre: server umount lustre-OST0000 complete [ 340.730757] Lustre: server umount lustre-OST0001 complete [ 342.523615] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 21:35:36 (1749605736) [ 348.605657] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 350.895404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 351.046643] LustreError: 8765:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.861307] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 354.002699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 354.860961] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 355.623700] Lustre: 10097:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 357.350295] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 357.353558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 358.699334] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 360.887754] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 360.892166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 362.149778] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 368.289210] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 369.084975] Lustre: 12402:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 378.971725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.977353] Lustre: Skipped 5 previous similar messages [ 381.025334] Lustre: server umount lustre-MDT0000 complete [ 381.926271] LustreError: 8749:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749605776 with bad export cookie 18254736784716621859 [ 381.927752] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 381.934482] LustreError: 8749:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 382.052790] Lustre: server umount lustre-MDT0001 complete [ 393.000819] Lustre: server umount lustre-OST0000 complete [ 404.084265] Lustre: server umount lustre-OST0001 complete [ 406.038757] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 21:36:39 (1749605799) [ 409.130560] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 411.257379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.371608] LustreError: 15069:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 412.069438] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 414.256977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 415.510548] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 416.581665] Lustre: 16398:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 418.979615] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 418.984807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 420.735150] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 421.141217] LustreError: 16869:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 423.886542] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 423.893477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 425.713983] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 428.994743] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 435.550921] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 436.579710] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (stopping) [ 436.581919] Lustre: Skipped 8 previous similar messages [ 439.371377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 442.403263] Lustre: server umount lustre-MDT0000 complete [ 443.454317] LustreError: 15053:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749605837 with bad export cookie 18254736784716622965 [ 443.455229] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 443.460946] LustreError: 15053:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 445.003587] LustreError: 15063:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 445.003894] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 445.003896] Lustre: Skipped 5 previous similar messages [ 445.021622] LustreError: 15063:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 449.610741] Lustre: server umount lustre-MDT0001 complete [ 452.795663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 453.013395] LustreError: 3756:0:(client.c:1383:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88009f4b9880 x1834594259468288/t0(0) o250->MGC192.168.203.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 453.226126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 453.230374] Lustre: Skipped 8 previous similar messages [ 454.415089] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 457.562797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 458.846258] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 460.625340] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 473.735866] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 477.533501] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec [ 478.664334] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 478.748117] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 478.751702] Lustre: Skipped 1 previous similar message [ 479.244078] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 482.731948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 486.648979] LustreError: 21745:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 486.653053] LustreError: 21745:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 486.679164] Lustre: server umount lustre-MDT0000 complete [ 488.092222] LustreError: 15053:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749605882 with bad export cookie 18254736784716624106 [ 488.094178] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 488.105681] LustreError: 15053:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 488.248383] Lustre: server umount lustre-MDT0001 complete [ 499.690077] Lustre: server umount lustre-OST0000 complete [ 510.836259] Lustre: server umount lustre-OST0001 complete [ 513.739501] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 21:38:27 (1749605907) [ 518.600673] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 521.809169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.976668] LustreError: 24216:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.984357] LustreError: 24216:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 523.166458] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 526.308037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 527.637590] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 528.686277] Lustre: 25555:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 528.688652] Lustre: 25555:0:(mgs_llog.c:1347:mgs_modify_param()) Skipped 1 previous similar message [ 531.150310] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 531.156208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 533.121499] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 536.247629] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 536.254683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 538.186098] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 541.389123] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 547.990876] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 558.225612] Lustre: server umount lustre-MDT0000 complete [ 559.583838] LustreError: 27032:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749605953 with bad export cookie 18254736784716624918 [ 559.585643] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 559.596608] LustreError: 27032:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 559.746145] Lustre: server umount lustre-MDT0001 complete [ 571.432745] Lustre: server umount lustre-OST0000 complete [ 582.927605] Lustre: server umount lustre-OST0001 complete [ 585.702552] LustreError: 29368:0:(class_obd.c:478:class_handle_ioctl()) (null): iocontrol from 'lctl' cmd=400866bb _IOW('f', 187, 8) no device found: rc = -22 [ 586.292581] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 21:39:39 (1749605979) [ 587.151782] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 587.756032] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 21:39:41 (1749605981) [ 591.106578] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 593.277441] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 596.257611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 598.457821] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 598.464580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 600.948577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 600.976765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.101131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 601.156926] Lustre: lustre-MDT0000: new disk, initializing [ 601.195255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 602.335574] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 605.507468] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 608.023497] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 608.029390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 608.048487] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 608.053734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 609.958747] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 610.095878] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 610.109340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 613.146056] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 613.903130] Lustre: DEBUG MARKER: create test files [ 625.099650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 625.106903] Lustre: Skipped 9 previous similar messages [ 625.110018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 625.113745] Lustre: Skipped 22 previous similar messages [ 628.989756] Lustre: server umount lustre-OST0000 complete [ 630.489585] Lustre: server umount lustre-MDT0000 complete [ 634.324181] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 635.120763] Key type lgssc unregistered [ 635.231144] LNet: 2253:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.240164] LNet: Removed LNI 192.168.203.144@tcp [ 646.947730] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 647.081581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 647.085684] alg: No test for adler32 (adler32-zlib) [ 647.912875] Lustre: Lustre: Build Version: 2.16.56_1_g45307c0 [ 647.989381] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 647.993037] LNet: Accept secure, port 988 [ 649.539698] Key type lgssc registered [ 649.870584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 652.646935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 652.660954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.810539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 654.967097] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 656.915239] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 659.461614] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 659.470039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 659.643813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 661.563200] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 663.538155] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 664.625032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 664.659409] Lustre: 5303:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 664.665827] Lustre: 5303:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 664.670632] Lustre: 5303:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 664.675780] Lustre: 5303:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 664.681154] Lustre: 5303:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 664.686129] Lustre: 5303:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 671.639413] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 673.456646] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 679.643574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 679.651577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 681.338613] LustreError: 6230:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 681.414560] Lustre: server umount lustre-OST0000 complete [ 682.722714] LustreError: 6501:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 682.727235] LustreError: 6501:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 682.805367] Lustre: server umount lustre-MDT0000 complete [ 686.771856] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 687.623011] Key type lgssc unregistered [ 687.738291] LNet: 7027:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.746077] LNet: Removed LNI 192.168.203.144@tcp [ 700.357148] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 700.533576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 700.557765] alg: No test for adler32 (adler32-zlib) [ 701.406987] Lustre: Lustre: Build Version: 2.16.56_1_g45307c0 [ 701.486101] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 701.489308] LNet: Accept secure, port 988 [ 703.037733] Key type lgssc registered [ 703.393958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 706.312887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 706.324181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.476710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 708.638837] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 710.505632] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 713.092270] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 713.099179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 713.268348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.122290] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 717.005847] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 717.269753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 717.297686] Lustre: 10197:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 717.303124] Lustre: 10197:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 717.307402] Lustre: 10197:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 717.312029] Lustre: 10197:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 717.316756] Lustre: 10197:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 717.321194] Lustre: 10197:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 725.102270] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 729.447614] LustreError: 11054:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 729.527774] Lustre: server umount lustre-OST0000 complete [ 740.907185] LustreError: 11322:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 740.909380] LustreError: 11322:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 740.969992] Lustre: server umount lustre-MDT0000 complete [ 744.211816] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 745.012999] Key type lgssc unregistered [ 745.123142] LNet: 11854:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.131086] LNet: Removed LNI 192.168.203.144@tcp [ 752.519004] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 758.369089] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 758.520593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 758.524425] alg: No test for adler32 (adler32-zlib) [ 759.360913] Lustre: Lustre: Build Version: 2.16.56_1_g45307c0 [ 759.437634] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 759.440844] LNet: Accept secure, port 988 [ 760.989689] Key type lgssc registered [ 761.339032] Lustre: Echo OBD driver; http://www.lustre.org/ [ 764.655349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 767.173409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 769.441773] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 769.450755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 771.778095] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 771.785825] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 774.345013] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 774.366239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 774.375447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 775.457348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 775.474582] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 775.530926] Lustre: lustre-MDT0000: new disk, initializing [ 775.563097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 775.573743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 776.724887] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 781.166937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 781.189076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 781.236075] Lustre: 15580:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 781.267347] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 781.271784] Lustre: Skipped 1 previous similar message [ 781.325832] Lustre: lustre-MDT0001: new disk, initializing [ 781.362925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 781.380968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 781.386656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 782.551373] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 785.807831] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 787.108692] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 789.663157] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 789.669572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 789.692832] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 789.700551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 789.818382] Lustre: lustre-OST0000: new disk, initializing [ 789.821436] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 789.852093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 791.715567] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 794.989195] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 796.238758] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 797.078931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 797.084861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 797.117186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 797.348686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 798.577188] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 798.658574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 802.107990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 802.108407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 802.109279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 802.123677] LustreError: Skipped 1 previous similar message [ 805.373568] LustreError: 19130:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 805.453508] Lustre: server umount lustre-OST0000 complete [ 807.115437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 807.116143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 807.116145] Lustre: Skipped 1 previous similar message [ 807.130376] Lustre: Skipped 2 previous similar messages [ 812.123417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 812.127224] Lustre: Skipped 2 previous similar messages [ 812.959617] LustreError: 19401:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 812.963570] LustreError: 19401:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 812.989268] Lustre: server umount lustre-MDT0000 complete [ 814.386505] LustreError: 15572:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749606208 with bad export cookie 18155339728012275037 [ 814.387896] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 814.397132] LustreError: 15572:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 814.464860] LustreError: 19672:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 814.468441] LustreError: 19672:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 814.539413] Lustre: server umount lustre-MDT0001 complete [ 816.353460] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 21:43:30 (1749606210) [ 818.484015] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 821.787038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 830.613150] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 833.472870] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 836.648915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 839.056617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 841.363792] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 841.368043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 843.683922] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 843.692987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 846.301987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 846.330564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 846.452238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 846.470285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 846.528799] Lustre: lustre-MDT0000: new disk, initializing [ 846.561791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 846.573178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 847.836062] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 852.077977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 852.097833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 852.136173] Lustre: 24896:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 852.162327] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 852.166970] Lustre: Skipped 1 previous similar message [ 852.217617] Lustre: lustre-MDT0001: new disk, initializing [ 852.250309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 852.267399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 852.272637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 853.283401] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 856.525170] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 857.774118] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 860.316049] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 860.324566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 860.348747] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 860.354107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 860.479083] Lustre: lustre-OST0000: new disk, initializing [ 860.483389] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 860.508593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 861.754555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 861.760424] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 861.787331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 862.394705] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 865.547730] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 866.802902] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 866.895057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 868.007079] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 868.082665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 870.835416] LustreError: 28439:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 870.839617] LustreError: 28439:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 870.914108] Lustre: server umount lustre-OST0000 complete [ 876.795613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 876.796117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 876.807004] Lustre: Skipped 1 previous similar message [ 881.803382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 881.807073] Lustre: Skipped 2 previous similar messages [ 882.806956] LustreError: 28706:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 882.809350] LustreError: 28706:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 882.835401] Lustre: server umount lustre-MDT0000 complete [ 884.267837] LustreError: 24889:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749606278 with bad export cookie 18155339728012275905 [ 884.269770] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 884.281060] LustreError: 24889:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 894.276688] LustreError: 28981:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 894.280595] LustreError: 28981:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 894.383225] Lustre: server umount lustre-MDT0001 complete [ 897.267205] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 21:44:50 (1749606290) [ 899.690460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.902026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 901.046466] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 904.219697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 905.591336] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 907.598570] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 908.841169] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 911.299075] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 911.307210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 913.274888] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 915.137192] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 916.479367] Lustre: 32761:0:(genops.c:1791:obd_export_evict_by_uuid()) MGS: evicting a2e42fba-96fc-41ed-810d-1a5f769a8fba at adminstrative request [ 917.349988] LustreError: MGC192.168.203.144@tcp: operation ldlm_enqueue to node 0@lo failed: rc = -107 [ 917.354757] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 917.361224] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 917.361267] Lustre: 29862:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 927.363704] Lustre: MGS: Client a2e42fba-96fc-41ed-810d-1a5f769a8fba (at 0@lo) reconnecting [ 927.368623] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xfbf4b94aa0b57425 to 0xfbf4b94aa0b5769b [ 927.376385] Lustre: MGC192.168.203.144@tcp: Connection restored to (at 0@lo) [ 948.986271] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 948.993300] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 949.016157] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 949.022266] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 949.080165] Lustre: lustre-OST0001: new disk, initializing [ 949.083478] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 949.107430] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 949.111445] Lustre: Skipped 2 previous similar messages [ 950.966009] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 954.179275] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 955.248620] LustreError: 1756:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 955.252544] LustreError: 1756:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 955.267500] Lustre: server umount lustre-OST0001 complete [ 955.937110] LustreError: 31852:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 955.944988] LustreError: 31852:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 959.531296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 959.536382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 959.544870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 960.972986] LustreError: 31851:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 960.982313] LustreError: 31851:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 962.738008] Lustre: server umount lustre-OST0000 complete [ 964.443405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 964.448044] LustreError: Skipped 1 previous similar message [ 964.450806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.458325] Lustre: Skipped 1 previous similar message [ 970.208610] Lustre: server umount lustre-MDT0000 complete [ 970.987343] LustreError: 29868:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 970.997328] LustreError: 29868:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 971.562528] LustreError: 829:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749606365 with bad export cookie 18155339728012277403 [ 971.564003] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 971.571510] LustreError: 829:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 971.672593] Lustre: server umount lustre-MDT0001 complete [ 974.482299] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 21:46:08 (1749606368) [ 975.012904] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 975.638559] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 21:46:09 (1749606369) [ 976.220094] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 976.862651] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 21:46:10 (1749606370) [ 977.467972] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 978.129265] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 21:46:11 (1749606371) [ 978.723188] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 980.623172] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 980.629346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 981.015882] loop: module loaded [ 981.021602] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 981.030565] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 981.607313] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 982.460953] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 983.512971] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 986.556345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 987.994681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 989.391026] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 989.397584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 990.803114] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 990.809540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 993.356019] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 993.361943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 993.382652] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 993.388588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 993.399169] LustreError: 5915:0:(obd_config.c:779:class_setup()) setup lustre-OST0000-osd failed (-28) [ 993.403759] LustreError: 5915:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 993.408522] LustreError: 5915:0:(tgt_mount.c:2236:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 993.414276] LustreError: 5915:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 999.951322] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 1002.708670] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1005.920984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1008.408053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1010.580616] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1010.587438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1012.989537] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1012.997011] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1015.536198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1015.560778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.683677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1015.700435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1015.758360] Lustre: lustre-MDT0000: new disk, initializing [ 1015.789043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1015.799624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1016.962380] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1021.209304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1021.235344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1021.276413] Lustre: 9449:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1021.405547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1022.417975] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1025.581990] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1026.822671] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1029.305478] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1029.314391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1029.337312] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1029.342574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1030.052373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1030.099959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1031.345954] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1034.593070] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1035.782983] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1035.860965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1036.996033] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1037.076562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1039.794642] LustreError: 12985:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1039.798242] LustreError: 12985:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1039.875243] Lustre: server umount lustre-OST0000 complete [ 1041.403501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1041.408086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1041.414862] Lustre: Skipped 1 previous similar message [ 1041.418034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1041.421487] Lustre: Skipped 6 previous similar messages [ 1051.131422] Lustre: server umount lustre-MDT0000 complete [ 1052.499956] LustreError: 9441:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749606446 with bad export cookie 18155339728012277998 [ 1052.502070] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1052.511949] LustreError: 9441:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1062.592147] Lustre: server umount lustre-MDT0001 complete [ 1065.529042] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 21:47:39 (1749606459) [ 1067.770427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1067.966318] LustreError: 14431:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1069.103837] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1072.195332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1073.524261] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1075.457460] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1076.655116] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1079.197266] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1079.203312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1079.380047] mount.lustre (16404) used greatest stack depth: 9264 bytes left [ 1081.213245] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1083.114167] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1084.327153] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1093.625970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 1094.826110] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1094.913948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1097.400141] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1097.406466] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1097.436101] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1097.442219] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1097.503256] Lustre: lustre-OST0001: new disk, initializing [ 1097.506193] Lustre: Skipped 2 previous similar messages [ 1097.509810] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1097.514215] Lustre: Skipped 3 previous similar messages [ 1099.375287] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1102.639957] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1103.467824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1103.476071] Lustre: Skipped 2 previous similar messages [ 1103.480175] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1103.508293] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1103.773260] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1103.840808] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1105.027279] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1105.108848] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1115.203165] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1123.139881] Lustre: Failing over lustre-MDT0000 [ 1123.282184] LustreError: 20442:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1123.285389] LustreError: 20442:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 1123.309804] Lustre: server umount lustre-MDT0000 complete [ 1123.516315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1123.517631] LustreError: 14453:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1123.533561] Lustre: Skipped 4 previous similar messages [ 1133.531661] LustreError: 16051:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1133.540985] LustreError: 16051:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 14 previous similar messages [ 1138.400251] LDISKFS-fs (dm-0): recovery complete [ 1138.403073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1138.455900] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1138.591014] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1139.723001] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1141.437612] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1141.437643] Lustre: 14426:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1143.579940] Lustre: 16051:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1143.586061] Lustre: 16051:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 1 previous similar message [ 1143.592303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1143.611557] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1143.920630] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1144.328465] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1144.333838] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 1 previous similar message [ 1144.638606] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1144.643336] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 1 previous similar message [ 1144.651064] Lustre: 21400:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1144.658461] Lustre: 21400:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 7 previous similar messages [ 1145.367650] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1145.375703] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 2 previous similar messages [ 1145.683629] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1145.687898] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 2 previous similar messages [ 1146.898862] Lustre: 21400:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1146.909625] Lustre: 21400:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 5 previous similar messages [ 1147.718740] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1147.723393] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 4 previous similar messages [ 1147.729363] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1147.735138] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 5 previous similar messages [ 1151.150814] Lustre: 21400:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1151.156814] Lustre: 21400:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 10 previous similar messages [ 1151.961608] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1151.964632] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 10 previous similar messages [ 1151.969532] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1151.974445] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 10 previous similar messages [ 1159.403781] Lustre: 21400:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1159.410595] Lustre: 21400:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 20 previous similar messages [ 1160.217611] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1160.222613] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 20 previous similar messages [ 1160.228779] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1160.234647] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 20 previous similar messages [ 1175.799743] Lustre: 21400:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1175.806597] Lustre: 21400:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 41 previous similar messages [ 1176.613684] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1176.617087] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 41 previous similar messages [ 1176.622116] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1176.626139] LustreError: 21400:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 41 previous similar messages [ 1200.728638] Lustre: 21400:0:(ldlm_lib.c:2309:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1200.733451] Lustre: 21400:0:(ldlm_lib.c:2319:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1200.737994] Lustre: 21400:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 2c57d8a9-8265-48b5-b54f-cccebb53ab98@192.168.203.44@tcp [ 1200.744775] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1200.748621] Lustre: 21400:0:(ldlm_lib.c:1831:abort_req_replay_queue()) @@@ aborted: req@ffff88009d3af480 x1834595015255296/t0(8589934746) o35->2c57d8a9-8265-48b5-b54f-cccebb53ab98@192.168.203.44@tcp:560/0 lens 392/0 e 0 to 0 dl 1749606605 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 1200.762724] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1200.768668] Lustre: lustre-MDT0000: Denying connection for new client 2c57d8a9-8265-48b5-b54f-cccebb53ab98 (at 192.168.203.44@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 1200.772979] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1200.782108] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1200.782111] Lustre: Skipped 2 previous similar messages [ 1200.782670] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1200.803662] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:39 to 0x2c0000401:545) [ 1200.804097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:72 to 0x280000401:577) [ 1207.563850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1207.568693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1207.577162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1207.581055] Lustre: Skipped 4 previous similar messages [ 1213.314574] Lustre: server umount lustre-OST0000 complete [ 1213.692170] LustreError: 16411:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.700884] LustreError: 16411:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1220.717137] Lustre: server umount lustre-OST0001 complete [ 1228.251388] Lustre: server umount lustre-MDT0000 complete [ 1229.618951] LustreError: 18138:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749606623 with bad export cookie 18155339728012329126 [ 1229.621064] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1229.631150] LustreError: 18138:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1229.775021] Lustre: server umount lustre-MDT0001 complete [ 1232.729998] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 21:50:26 (1749606626) [ 1235.516846] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1235.524724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1235.529722] Lustre: *** cfs_fail_loc=197, val=0*** [ 1252.635374] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1258.641968] LustreError: 23957:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.144@tcp: failed processing log, type 4: rc = -110 [ 1264.658374] Lustre: *** cfs_fail_loc=197, val=0*** [ 1264.660835] Lustre: Skipped 65 previous similar messages [ 1287.676816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1287.683733] Lustre: Skipped 7 previous similar messages [ 1289.515830] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1291.374383] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1292.106376] LustreError: 24800:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1292.110192] LustreError: 24800:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 1292.190208] Lustre: server umount lustre-OST0000 complete [ 1295.066344] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 21:51:28 (1749606688) [ 1308.774850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1308.957756] LustreError: 25891:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1308.964154] LustreError: 25891:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 1310.084485] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1313.271057] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1315.343782] LustreError: 25923:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1315.350282] LustreError: 25923:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1315.355792] LustreError: 25923:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1315.527242] Lustre: server umount lustre-MDT0000 complete [ 1320.907154] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1320.913297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1326.541636] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 1329.271706] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1332.293444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1334.693025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1336.912853] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1336.918530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1339.190183] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1339.196197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1341.685054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1341.714827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.815423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1341.828155] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1341.886032] Lustre: lustre-MDT0000: new disk, initializing [ 1341.922911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1343.076484] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1347.442537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1347.469445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1347.510018] Lustre: 30973:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1347.632425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1348.742332] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1351.902340] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1353.127704] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1355.523804] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1355.529774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1355.550527] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1355.555812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1357.422792] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1357.526575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1360.575183] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1361.786269] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1361.863681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1362.999134] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1363.076281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1367.516045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1367.516102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1367.516105] LustreError: Skipped 2 previous similar messages [ 1367.516587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1367.516588] Lustre: Skipped 7 previous similar messages [ 1367.536017] Lustre: Skipped 6 previous similar messages [ 1369.822993] Lustre: server umount lustre-OST0000 complete [ 1377.319250] Lustre: server umount lustre-MDT0000 complete [ 1377.531324] LustreError: 32320:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1377.543224] LustreError: 32320:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1378.684908] LustreError: 30966:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749606772 with bad export cookie 18155339728012339437 [ 1378.687130] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1378.697329] LustreError: 30966:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1378.836069] Lustre: server umount lustre-MDT0001 complete [ 1381.579989] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 21:52:55 (1749606775) [ 1384.728936] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1385.502285] Key type lgssc unregistered [ 1385.608165] LNet: 3411:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1386.615076] LNet: Removed LNI 192.168.203.144@tcp [ 1392.380652] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 1398.023118] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1398.220783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1398.224670] alg: No test for adler32 (adler32-zlib) [ 1399.075078] Lustre: Lustre: Build Version: 2.16.56_1_g45307c0 [ 1399.158675] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1399.162051] LNet: Accept secure, port 988 [ 1400.708695] Key type lgssc registered [ 1400.935114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1403.753878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1405.649296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1407.623120] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1407.629043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1409.818709] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1409.823570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1411.526484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1413.708883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1413.732013] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1413.738646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1414.808058] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1414.817477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1414.865015] Lustre: lustre-MDT0000: new disk, initializing [ 1414.887862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1414.895800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1415.924568] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1420.418226] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1420.424277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1420.446745] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1420.452036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1420.572582] Lustre: lustre-OST0000: new disk, initializing [ 1420.576651] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1420.580927] Lustre: Skipped 1 previous similar message [ 1420.608997] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1422.422586] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1423.207123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1423.212576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1423.225759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1426.840339] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1426.846462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1426.871080] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1426.876285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1426.929303] Lustre: lustre-OST0001: new disk, initializing [ 1426.932706] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1426.961702] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1428.764741] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1430.139815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 1430.142831] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 1430.152710] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 1435.147573] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1435.148550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1435.158153] Lustre: Skipped 1 previous similar message [ 1438.752142] LustreError: 10171:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 1438.835730] Lustre: server umount lustre-MDT0000 complete [ 1440.838782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1442.505865] LustreError: 10172:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749606836 with bad export cookie 17479932596570971187 [ 1442.508251] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1442.518129] LustreError: 10172:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1442.536929] LustreError: 10965:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1442.541355] LustreError: 10965:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1442.555891] Lustre: server umount lustre-OST0000 complete [ 1443.958434] LustreError: 11230:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1443.962196] LustreError: 11230:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1444.036767] Lustre: server umount lustre-OST0001 complete [ 1446.522211] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 21:54:00 (1749606840) [ 1448.721302] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1451.023172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1451.303051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1453.093180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1453.461678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1455.625239] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 21:54:09 (1749606849) [ 1460.057782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1460.761107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1461.489831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1462.190797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1462.551397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1463.279877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1468.801902] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 1471.454510] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1474.591110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1476.884416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1479.085416] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1479.091689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1481.362630] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1481.368833] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1483.863864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1483.891073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1483.990311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1484.005600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1484.051896] Lustre: lustre-MDT0000: new disk, initializing [ 1484.078485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1484.087341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1485.055565] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1488.933169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1488.950296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1488.977759] Lustre: 17880:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1488.991859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1488.993555] Lustre: Skipped 1 previous similar message [ 1489.025870] Lustre: lustre-MDT0001: new disk, initializing [ 1489.045010] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1489.054353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1489.057246] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1489.879866] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1492.639690] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1493.586869] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1495.629753] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1495.635224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1495.653694] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1495.658057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1497.158716] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1499.259199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1499.264538] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1499.292582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1500.034314] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1501.165746] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1501.239570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1502.194552] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1502.261566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1504.283664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1504.283924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1504.284373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1504.284375] Lustre: Skipped 1 previous similar message [ 1504.300156] Lustre: Skipped 1 previous similar message [ 1504.942601] LustreError: 21416:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1504.947117] LustreError: 21416:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1505.012067] Lustre: server umount lustre-OST0000 complete [ 1509.067811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1509.072068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1509.079030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1509.082500] Lustre: Skipped 1 previous similar message [ 1512.274313] LustreError: 21683:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1512.278820] LustreError: 21683:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1512.311558] Lustre: server umount lustre-MDT0000 complete [ 1513.524356] LustreError: 17872:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749606907 with bad export cookie 17479932596570972398 [ 1513.525354] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1513.530209] LustreError: 17872:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1513.647957] Lustre: server umount lustre-MDT0001 complete [ 1516.078524] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 21:55:09 (1749606909) [ 1518.205982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1518.355356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1518.357133] Lustre: Skipped 1 previous similar message [ 1519.153010] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1521.970229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1523.121252] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1524.506249] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1525.471195] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1527.560430] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1527.565006] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1529.106030] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1530.634826] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1536.110303] Lustre: *** cfs_fail_loc=159, val=0*** [ 1552.113367] Lustre: lustre-MDT0000: Client 844d17ce-266e-4f6c-a3d4-9b3942e2a89c (at 192.168.203.44@tcp) reconnecting [ 1552.395463] Lustre: *** cfs_fail_loc=159, val=0*** [ 1552.397466] Lustre: Skipped 5 previous similar messages [ 1568.401456] Lustre: lustre-MDT0000: Client 844d17ce-266e-4f6c-a3d4-9b3942e2a89c (at 192.168.203.44@tcp) reconnecting [ 1572.747624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1572.748380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1572.748403] Lustre: Skipped 1 previous similar message [ 1572.749035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1572.749037] Lustre: Skipped 1 previous similar message [ 1575.071484] LustreError: 25891:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1575.075396] LustreError: 25891:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1575.155984] Lustre: server umount lustre-OST0000 complete [ 1577.163102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1577.167092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1577.173365] Lustre: Skipped 1 previous similar message [ 1577.176160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1577.177825] Lustre: Skipped 1 previous similar message [ 1582.528042] Lustre: server umount lustre-MDT0000 complete [ 1582.762989] LustreError: 22833:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1582.766781] LustreError: 22833:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1583.494018] LustreError: 22816:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749606977 with bad export cookie 17479932596570973266 [ 1583.496375] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1583.506794] LustreError: 22816:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1583.649890] Lustre: server umount lustre-MDT0001 complete [ 1587.401730] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1588.161566] Key type lgssc unregistered [ 1588.259877] LNet: 26959:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.265122] LNet: Removed LNI 192.168.203.144@tcp [ 1598.125176] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1598.281193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1598.299434] alg: No test for adler32 (adler32-zlib) [ 1599.154189] Lustre: Lustre: Build Version: 2.16.56_1_g45307c0 [ 1599.240384] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1599.244468] LNet: Accept secure, port 988 [ 1600.794711] Key type lgssc registered [ 1601.137348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1603.976212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1603.985778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1605.190263] LustreError: 28482:0:(ldlm_lib.c:523:client_obd_setup()) can't add initial connection [ 1605.194065] LustreError: 28482:0:(osp_dev.c:1207:osp_init0()) lustre-MDT0001-osp-MDT0000: can't setup obd: rc = -2 [ 1605.198535] LustreError: 28482:0:(obd_config.c:779:class_setup()) setup lustre-MDT0001-osp-MDT0000 failed (-2) [ 1605.202853] LustreError: 28482:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -2 [ 1605.208192] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.144@tcp [ 1605.213905] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1605.223590] LustreError: 28445:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1605.229471] LustreError: 28445:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -2 [ 1605.234550] Lustre: Failing over lustre-MDT0000 [ 1605.311569] LustreError: 28445:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1605.384762] Lustre: server umount lustre-MDT0000 complete [ 1605.387010] LustreError: 28445:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1605.961777] Lustre: DEBUG MARKER: conf-sanity test_90b: @@@@@@ FAIL: MDT start failed [ 1607.847282] Lustre: DEBUG MARKER: == conf-sanity test 90c: check max_mod_rpcs_in_flight update limits ========================================================== 21:56:41 (1749607001) [ 1610.289382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1610.420159] LustreError: 29309:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 1610.425528] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1610.429592] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1610.433623] LustreError: 29272:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 1610.436569] LustreError: 29272:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -17 [ 1610.440800] Lustre: Failing over lustre-MDT0000 [ 1610.506066] LustreError: 29272:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1610.511229] LustreError: 29272:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1610.575018] Lustre: server umount lustre-MDT0000 complete [ 1610.576661] LustreError: 29272:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 1611.125883] Lustre: DEBUG MARKER: conf-sanity test_90c: @@@@@@ FAIL: MDT start failed [ 1612.941585] Lustre: DEBUG MARKER: == conf-sanity test 90d: check one close RPC is allowed above max_mod_rpcs_in_flight ========================================================== 21:56:46 (1749607006) [ 1615.278972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1615.437395] LustreError: 30133:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 1615.443467] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1615.447949] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1615.455593] LustreError: 30096:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 1615.459896] LustreError: 30096:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -17 [ 1615.464005] Lustre: Failing over lustre-MDT0000 [ 1615.542236] LustreError: 30096:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1615.546237] LustreError: 30096:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1615.610543] Lustre: server umount lustre-MDT0000 complete [ 1615.612468] LustreError: 30096:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 1616.160912] Lustre: DEBUG MARKER: conf-sanity test_90d: @@@@@@ FAIL: MDT start failed [ 1617.965770] Lustre: DEBUG MARKER: == conf-sanity test 91: evict-by-nid support ============= 21:56:51 (1749607011) [ 1620.439402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1620.603355] LustreError: 30958:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 1620.609018] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1620.613912] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1620.621982] LustreError: 30920:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 1620.626554] LustreError: 30920:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -17 [ 1620.629651] Lustre: Failing over lustre-MDT0000 [ 1620.696685] LustreError: 30920:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1620.699360] LustreError: 30920:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1620.754639] Lustre: server umount lustre-MDT0000 complete [ 1620.756378] LustreError: 30920:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 1621.305124] Lustre: DEBUG MARKER: conf-sanity test_91: @@@@@@ FAIL: MDS start failed [ 1623.090074] Lustre: DEBUG MARKER: == conf-sanity test 92: ldev returns MGS NID correctly in command substitution ========================================================== 21:56:56 (1749607016) [ 1626.270175] Lustre: DEBUG MARKER: == conf-sanity test 93: register mulitple MDT at the same time ========================================================== 21:56:59 (1749607019) [ 1626.792551] Lustre: DEBUG MARKER: SKIP: conf-sanity test_93 needs >= 3 MDTs [ 1627.395352] Lustre: DEBUG MARKER: == conf-sanity test 94: ldev outputs correct labels for file system name query ========================================================== 21:57:00 (1749607020) [ 1630.097534] Lustre: DEBUG MARKER: == conf-sanity test 95: ldev should only allow one label filter ========================================================== 21:57:03 (1749607023) [ 1633.099839] Lustre: DEBUG MARKER: == conf-sanity test 96: ldev returns hostname and backend fs correctly in command sub ========================================================== 21:57:06 (1749607026) [ 1635.816981] Lustre: DEBUG MARKER: == conf-sanity test 97: ldev returns correct ouput when querying based on role ========================================================== 21:57:09 (1749607029) [ 1638.503664] Lustre: DEBUG MARKER: == conf-sanity test 98: Buffer-overflow check while parsing mount_opts ========================================================== 21:57:11 (1749607031) [ 1640.903367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1641.069127] LustreError: 758:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 1641.072402] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1641.075528] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1641.083912] LustreError: 721:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 1641.088836] LustreError: 721:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -17 [ 1641.092983] Lustre: Failing over lustre-MDT0000 [ 1641.173193] LustreError: 721:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1641.177339] LustreError: 721:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1641.242579] Lustre: server umount lustre-MDT0000 complete [ 1641.244616] LustreError: 721:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 1641.793172] Lustre: DEBUG MARKER: conf-sanity test_98: @@@@@@ FAIL: MDT start failed [ 1643.376158] Lustre: DEBUG MARKER: == conf-sanity test 99: Adding meta_bg option ============ 21:57:17 (1749607037) [ 1645.608380] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1647.676057] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1647.684031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1650.217203] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 21:57:23 (1749607043) [ 1655.340300] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 1656.969761] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1659.193101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1660.755265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1662.195323] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1662.198988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1663.722904] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1663.725657] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1665.529477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1665.546230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1665.605177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1665.613190] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1665.649775] Lustre: lustre-MDT0000: new disk, initializing [ 1665.669392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1665.675237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1666.685038] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1671.077257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1671.101697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1671.148211] Lustre: 5548:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1671.176406] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1671.180403] Lustre: Skipped 1 previous similar message [ 1671.229794] Lustre: lustre-MDT0001: new disk, initializing [ 1671.261492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1671.275206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1671.280435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1671.484992] LustreError: 6554:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 1671.490902] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1671.496170] LustreError: 5550:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1672.405436] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1675.503236] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1676.712974] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1678.731079] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1678.739754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1678.756929] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1678.762394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1678.873843] Lustre: lustre-OST0000: new disk, initializing [ 1678.876740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1678.906413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1680.230949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1680.238575] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1680.279330] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1680.788104] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1683.996848] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1690.267105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1690.269652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1690.273984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1693.730906] LustreError: 8754:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1693.733426] LustreError: 8754:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1693.785316] Lustre: server umount lustre-OST0000 complete [ 1695.291451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1695.292012] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1695.292015] Lustre: Skipped 1 previous similar message [ 1695.305874] Lustre: Skipped 2 previous similar messages [ 1700.299368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1700.302126] Lustre: Skipped 2 previous similar messages [ 1700.917430] Lustre: server umount lustre-MDT0000 complete [ 1701.922207] LustreError: 5539:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749607096 with bad export cookie 10603547655939856352 [ 1701.924238] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1701.929442] LustreError: 5539:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1702.032644] Lustre: server umount lustre-MDT0001 complete [ 1704.888422] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1705.615801] Key type lgssc unregistered [ 1705.711088] LNet: 9821:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1706.718964] LNet: Removed LNI 192.168.203.144@tcp [ 1713.861910] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1713.957790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1713.961947] alg: No test for adler32 (adler32-zlib) [ 1714.766900] Lustre: Lustre: Build Version: 2.16.56_1_g45307c0 [ 1714.809382] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1714.811416] LNet: Accept secure, port 988 [ 1716.335659] Key type lgssc registered [ 1716.501576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1718.412288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1718.418214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1719.531718] LustreError: 11343:0:(ldlm_lib.c:523:client_obd_setup()) can't add initial connection [ 1719.533748] LustreError: 11343:0:(osp_dev.c:1207:osp_init0()) lustre-MDT0001-osp-MDT0000: can't setup obd: rc = -2 [ 1719.536300] LustreError: 11343:0:(obd_config.c:779:class_setup()) setup lustre-MDT0001-osp-MDT0000 failed (-2) [ 1719.538694] LustreError: 11343:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -2 [ 1719.543695] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.144@tcp [ 1719.547488] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1719.551881] LustreError: 11306:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1719.554466] LustreError: 11306:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -2 [ 1719.556704] Lustre: Failing over lustre-MDT0000 [ 1719.605995] LustreError: 11306:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1719.651817] Lustre: server umount lustre-MDT0000 complete [ 1719.654188] LustreError: 11306:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1720.010629] Lustre: DEBUG MARKER: conf-sanity test_101a: @@@@@@ FAIL: MDT start failed [ 1721.292600] Lustre: DEBUG MARKER: == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 21:58:34 (1749607114) [ 1722.922233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1723.033303] LustreError: 12166:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 1723.035846] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1723.038098] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1723.041903] LustreError: 12129:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 1723.043983] LustreError: 12129:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -17 [ 1723.045972] Lustre: Failing over lustre-MDT0000 [ 1723.090139] LustreError: 12129:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1723.092726] LustreError: 12129:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1723.138340] Lustre: server umount lustre-MDT0000 complete [ 1723.139553] LustreError: 12129:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 1723.478326] Lustre: DEBUG MARKER: conf-sanity test_101b: @@@@@@ FAIL: MDT start failed [ 1724.546619] Lustre: DEBUG MARKER: SKIP: conf-sanity test_102 skipping excluded test 102 [ 1724.914231] Lustre: DEBUG MARKER: == conf-sanity test 103: rename filesystem name ========== 21:58:38 (1749607118) [ 1728.709771] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1731.562868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1731.749140] LustreError: 13825:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 1731.753562] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1731.756406] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1731.764013] LustreError: 13789:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 1731.767040] LustreError: 13789:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -17 [ 1731.771143] Lustre: Failing over lustre-MDT0000 [ 1731.840383] LustreError: 13789:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1731.844958] LustreError: 13789:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1731.908409] Lustre: server umount lustre-MDT0000 complete [ 1731.910170] LustreError: 13789:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 1734.251442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1751.350774] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1757.352612] LustreError: 14286:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.144@tcp: failed processing log, type 4: rc = -110 [ 1775.391723] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 1775.394930] Lustre: 14286:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 1779.392040] LustreError: 14305:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1779.398479] LustreError: 14305:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1784.394892] LustreError: 14305:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1787.404598] LustreError: 14301:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1787.409697] LustreError: 14301:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1788.280793] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 1788.283208] Lustre: 14298:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 1792.410954] LustreError: 14305:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1792.417721] LustreError: 14305:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1797.419124] LustreError: 14305:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1797.428572] LustreError: 14305:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1798.404692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1799.312763] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1800.375621] LustreError: 14820:0:(obd_class.h:1230:obd_iocontrol()) MGC192.168.203.144@tcp: no obd_iocontrol operation [ 1802.454244] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1802.460270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1809.435429] LustreError: 14305:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1809.446760] LustreError: 14305:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 1816.288116] LustreError: 14298:0:(mgc_request.c:614:do_requeue()) failed processing log: -110 [ 1822.292198] LustreError: 14298:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1828.296286] LustreError: 14298:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1829.460585] LustreError: 15305:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1829.468638] LustreError: 15305:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 7 previous similar messages [ 1848.563811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1850.181787] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1851.398017] LustreError: 14298:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1852.962199] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1852.967347] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1852.984989] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1852.989690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1859.595487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 1864.603245] LustreError: 14305:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1864.612381] LustreError: 14305:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 22 previous similar messages [ 1867.402245] Lustre: 14298:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 1873.409177] LustreError: 14298:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1874.619923] LustreError: lustre-MDT0000-lwp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 1884.015854] LustreError: lustre-OST0001: cannot register this server with the MGS: rc = -110. Is the MGS running? [ 1884.021160] LustreError: 16307:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -110 [ 1884.025683] LustreError: 16307:0:(tgt_mount.c:1762:server_put_super()) no obd lustre-OST0001 [ 1884.029956] LustreError: 16307:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0001 not registered [ 1884.036494] Lustre: server umount lustre-OST0001 complete [ 1884.039088] LustreError: 16307:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -110 [ 1885.414202] LustreError: 14298:0:(mgc_request.c:614:do_requeue()) failed processing log: -110 [ 1901.419081] LustreError: 14298:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1929.700344] LustreError: 14305:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1929.700808] LustreError: lustre-MDT0000-lwp-OST0000: operation mds_connect to node 0@lo failed: rc = -19 [ 1929.709792] LustreError: 14305:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 38 previous similar messages [ 1948.528271] LustreError: 14298:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1948.534355] LustreError: 14298:0:(mgc_request.c:614:do_requeue()) Skipped 2 previous similar messages [ 1996.539026] Lustre: 14298:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2008.827292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 2012.542137] LustreError: 14298:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2012.545994] LustreError: 14298:0:(mgc_request.c:614:do_requeue()) Skipped 2 previous similar messages [ 2023.852131] LustreError: lustre-MDT0000-lwp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 2058.907407] LustreError: 15305:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2058.912607] LustreError: 15305:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 77 previous similar messages [ 2078.932804] LustreError: lustre-MDT0000-lwp-OST0000: operation mds_connect to node 0@lo failed: rc = -19 [ 2151.752873] Lustre: 14298:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2159.067239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 2167.756930] LustreError: 14298:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2167.759566] LustreError: 14298:0:(mgc_request.c:614:do_requeue()) Skipped 7 previous similar messages [ 2210.781146] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2211.647530] LustreError: 17054:0:(obd_class.h:1230:obd_iocontrol()) MGC192.168.203.144@tcp: no obd_iocontrol operation [ 2216.997925] Lustre: DEBUG MARKER: conf-sanity test_103: @@@@@@ FAIL: Fail to mkdir /mnt/lustre/d103.conf-sanity. [ 2218.853448] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 22:06:52 (1749607612) [ 2220.261179] LustreError: 14382:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000009:0x0:0x0]: rc = -5 [ 2220.265468] LustreError: 14382:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't get id from catalogs: rc = -5 [ 2220.268383] LustreError: 14382:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0000-osp-MDT0001: get update log duration 422, retries 0, failed: rc = -5 [ 2220.318574] LustreError: 17778:0:(obd_class.h:479:obd_check_dev()) Device 6 not setup [ 2220.320889] LustreError: 17778:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2220.371990] Lustre: server umount lustre-MDT0001 complete [ 2256.614111] LustreError: 18044:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2256.616680] LustreError: 18044:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2256.671081] Lustre: server umount lustre-OST0000 complete [ 2258.820307] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 2261.037822] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2263.593733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2265.500192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2267.311366] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2267.316479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2269.183986] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2269.191292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2271.209827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2271.228835] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 2271.234531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 2271.325403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2271.344613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2271.382404] Lustre: lustre-MDT0000: new disk, initializing [ 2271.414779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2271.425643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2272.353637] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2276.143064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2276.163756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2276.189460] Lustre: 20930:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2276.207595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2276.210246] Lustre: Skipped 1 previous similar message [ 2276.242940] Lustre: lustre-MDT0001: new disk, initializing [ 2276.269286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2276.282362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2276.289291] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2277.151573] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2277.874770] LustreError: 22278:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 2277.879236] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2280.873163] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2280.877202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2280.905217] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2280.909924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2280.988694] Lustre: lustre-OST0000: new disk, initializing [ 2280.991326] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2281.013149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2282.402973] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2282.785276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2282.789813] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2282.826173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2286.173769] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2286.179444] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2286.204548] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2286.211015] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2286.259644] Lustre: lustre-OST0001: new disk, initializing [ 2286.262033] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2287.356960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2287.361608] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2287.377403] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2287.709201] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2292.400635] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 22:08:06 (1749607686) [ 2295.509754] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 22:08:09 (1749607689) [ 2307.404673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.406859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2307.412281] Lustre: Skipped 1 previous similar message [ 2312.411809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2312.414367] Lustre: Skipped 2 previous similar messages [ 2312.452665] LustreError: 25147:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2312.455968] LustreError: 25147:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2312.471810] Lustre: server umount lustre-OST0000 complete [ 2316.315113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2316.317668] LustreError: Skipped 1 previous similar message [ 2316.319318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.324605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2317.419399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2317.419718] LustreError: 22813:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.419720] LustreError: 22813:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 96 previous similar messages [ 2317.432758] Lustre: Skipped 1 previous similar message [ 2319.732926] Lustre: server umount lustre-MDT0000 complete [ 2320.707317] LustreError: 20922:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749607714 with bad export cookie 16434976777101136619 [ 2320.708349] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2320.714559] LustreError: 20922:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2322.427537] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2322.433623] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2322.438169] Lustre: Skipped 3 previous similar messages [ 2326.435123] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2326.816199] Lustre: server umount lustre-MDT0001 complete [ 2329.819341] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2338.712713] LustreError: 26771:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2338.715634] LustreError: 26771:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 2338.772869] Lustre: server umount lustre-OST0001 complete [ 2340.089490] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 2344.455993] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2346.600993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2348.251675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2349.714004] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2349.716528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2351.176775] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2351.180213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2352.879140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2352.895160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2352.964422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2352.975447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2353.013956] Lustre: lustre-MDT0000: new disk, initializing [ 2353.034191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2353.036559] Lustre: Skipped 1 previous similar message [ 2353.042184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2353.877328] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2357.342619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2357.361059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2357.385388] Lustre: 29436:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2357.461756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2358.280142] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2360.846660] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2361.718201] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2363.425754] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2363.429924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2363.445904] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2363.449757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2363.523640] Lustre: lustre-OST0000: new disk, initializing [ 2363.525215] Lustre: Skipped 1 previous similar message [ 2363.527338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2363.529277] Lustre: Skipped 2 previous similar messages [ 2364.633343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2364.636000] Lustre: Skipped 1 previous similar message [ 2364.637888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2364.655911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2364.725724] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2367.175960] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2369.659130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2369.659584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2369.664186] Lustre: Skipped 1 previous similar message [ 2375.076565] LustreError: 32444:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2375.079662] LustreError: 32444:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2375.127080] Lustre: server umount lustre-OST0000 complete [ 2379.675443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2379.679892] Lustre: Skipped 1 previous similar message [ 2382.159801] Lustre: server umount lustre-MDT0000 complete [ 2383.128442] LustreError: 31592:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749607777 with bad export cookie 16434976777101138089 [ 2383.129574] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.135527] LustreError: 31592:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2383.247270] Lustre: server umount lustre-MDT0001 complete [ 2386.199462] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2386.715375] Key type lgssc unregistered [ 2386.790822] LNet: 1078:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2387.795834] LNet: Removed LNI 192.168.203.144@tcp [ 2394.579263] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2394.687249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2394.715273] alg: No test for adler32 (adler32-zlib) [ 2395.530684] Lustre: Lustre: Build Version: 2.16.56_1_g45307c0 [ 2395.582444] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2395.583732] LNet: Accept secure, port 988 [ 2397.108682] Key type lgssc registered [ 2397.313110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2399.122777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2399.128167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2400.238794] LustreError: 2665:0:(ldlm_lib.c:523:client_obd_setup()) can't add initial connection [ 2400.240708] LustreError: 2665:0:(osp_dev.c:1207:osp_init0()) lustre-MDT0001-osp-MDT0000: can't setup obd: rc = -2 [ 2400.242890] LustreError: 2665:0:(obd_config.c:779:class_setup()) setup lustre-MDT0001-osp-MDT0000 failed (-2) [ 2400.244946] LustreError: 2665:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -2 [ 2400.247341] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.144@tcp [ 2400.249879] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2400.253757] LustreError: 2628:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2400.256009] LustreError: 2628:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -2 [ 2400.258132] Lustre: Failing over lustre-MDT0000 [ 2400.304341] LustreError: 2628:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2400.348666] Lustre: server umount lustre-MDT0000 complete [ 2400.350114] LustreError: 2628:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2400.700572] Lustre: DEBUG MARKER: conf-sanity test_107: @@@@@@ FAIL: start_mgsmds failed [ 2401.874154] Lustre: DEBUG MARKER: == conf-sanity test 108a: migrate from ldiskfs to ZFS ==== 22:09:55 (1749607795) [ 2402.210609] Lustre: DEBUG MARKER: SKIP: conf-sanity test_108a zfs only test [ 2402.595779] Lustre: DEBUG MARKER: == conf-sanity test 108b: migrate from ZFS to ldiskfs ==== 22:09:56 (1749607796) [ 2407.033175] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2410.330452] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2410.686893] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2410.974907] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 2410.977417] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2411.283986] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 2411.287087] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2411.521015] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2412.536076] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2413.479385] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 2413.482769] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2414.505293] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 2414.507402] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2415.488616] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2415.503329] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.515594] Lustre: 6543:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 2415.518907] Lustre: 6543:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 2415.522829] Lustre: 6543:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 2415.525859] Lustre: 6543:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 2415.530144] Lustre: 6543:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 2415.533704] Lustre: 6543:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 2415.537665] Lustre: 6543:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 2415.541808] Lustre: 6543:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 2415.546862] Lustre: 6543:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 2415.550596] Lustre: 6543:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 2415.554371] Lustre: 6543:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 2415.558265] Lustre: 6543:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 2415.562399] Lustre: 6543:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 2415.566038] Lustre: 6543:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 2415.796214] LustreError: 6628:0:(mgs_llog.c:1993:mgs_replace_nids()) Only MGS is allowed to be started [ 2415.799447] LustreError: 6628:0:(mgs_handler.c:952:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-MDT0000': rc = -115 [ 2416.453985] LustreError: 6822:0:(mgs_llog.c:1993:mgs_replace_nids()) Only MGS is allowed to be started [ 2416.455823] LustreError: 6822:0:(mgs_llog.c:1993:mgs_replace_nids()) Skipped 2 previous similar messages [ 2416.457512] LustreError: 6822:0:(mgs_handler.c:952:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0001': rc = -115 [ 2416.459525] LustreError: 6822:0:(mgs_handler.c:952:mgs_iocontrol()) Skipped 2 previous similar messages [ 2416.682124] LustreError: 6886:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2416.684085] LustreError: 6886:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2416.693984] Lustre: server umount MGS complete [ 2416.930765] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2417.051320] LustreError: 6991:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 2417.055053] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2417.057486] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2417.064153] LustreError: 6954:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 2417.066519] LustreError: 6954:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -17 [ 2417.068790] Lustre: Failing over lustre-MDT0000 [ 2417.179284] Lustre: server umount lustre-MDT0000 complete [ 2417.180598] LustreError: 6954:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 2417.536553] Lustre: DEBUG MARKER: conf-sanity test_108b: @@@@@@ FAIL: failed to mount mdt1 [ 2418.703551] Lustre: DEBUG MARKER: conf-sanity test_108b: @@@@@@ FAIL: failed to umount client [ 2419.779100] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 22:10:13 (1749607813) [ 2426.547526] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 2428.173669] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2430.293148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2432.024148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2433.621118] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2433.626357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2435.273915] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2435.277821] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2437.168640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2437.186807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2437.249007] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2437.259592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2437.296455] Lustre: lustre-MDT0000: new disk, initializing [ 2437.314412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2437.319952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2438.097258] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2441.599182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2441.626712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2441.655407] Lustre: 11717:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2441.674116] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2441.677040] Lustre: Skipped 1 previous similar message [ 2441.713046] Lustre: lustre-MDT0001: new disk, initializing [ 2441.738274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2441.748663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2441.751996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2442.586423] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2445.227632] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2446.102049] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2446.112538] LustreError: 13390:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 2446.115852] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2446.118893] LustreError: 11719:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2447.867733] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2447.871939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2447.892067] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2447.895434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2447.971053] Lustre: lustre-OST0000: new disk, initializing [ 2447.973172] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2447.992185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2449.200474] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2451.793639] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2453.928568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2453.931605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2453.958759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2460.954183] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 2463.177835] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 2471.227010] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 2479.677497] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 2519.051354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2519.051563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2519.052047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2519.059272] LustreError: Skipped 1 previous similar message [ 2521.841607] LustreError: 16246:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2521.843915] LustreError: 16246:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2521.895348] Lustre: server umount lustre-OST0000 complete [ 2524.059302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.059848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2524.059849] Lustre: Skipped 1 previous similar message [ 2524.066356] Lustre: Skipped 2 previous similar messages [ 2528.908523] LustreError: 16517:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2528.911158] LustreError: 16517:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2528.931946] Lustre: server umount lustre-MDT0000 complete [ 2529.884229] LustreError: 11709:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749607924 with bad export cookie 16356485200121213756 [ 2529.885182] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2529.892724] LustreError: 11709:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2530.004369] Lustre: server umount lustre-MDT0001 complete [ 2532.149317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2532.895750] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2534.133709] LustreError: 17968:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2534.136690] LustreError: 17968:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2534.151283] Lustre: server umount MGS complete [ 2537.045151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2537.187967] LustreError: 18784:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.194054] LustreError: 18796:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 2537.197452] LustreError: 18784:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2537.207603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.981636] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2540.238167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2541.130224] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2542.363116] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2543.174890] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2544.932229] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2544.935821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2545.027009] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2545.029228] Lustre: Skipped 1 previous similar message [ 2546.334672] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2547.680122] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2567.068337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2567.072627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2567.077658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2567.080053] Lustre: Skipped 1 previous similar message [ 2570.060211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2570.060213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2570.060226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2570.071344] Lustre: Skipped 1 previous similar message [ 2571.812702] LustreError: 22100:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2571.815218] LustreError: 22100:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2571.872497] Lustre: server umount lustre-OST0000 complete [ 2575.067806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2575.075234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2579.122284] Lustre: server umount lustre-MDT0000 complete [ 2580.074942] LustreError: 18801:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.080523] LustreError: 18801:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2580.176528] LustreError: 18766:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749607974 with bad export cookie 16356485200121215667 [ 2580.177552] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2580.184083] LustreError: 18766:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2580.310982] Lustre: server umount lustre-MDT0001 complete [ 2583.510834] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2584.139828] Key type lgssc unregistered [ 2584.240024] LNet: 23165:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2585.245847] LNet: Removed LNI 192.168.203.144@tcp [ 2594.286883] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 2598.817934] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2598.987329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2598.991589] alg: No test for adler32 (adler32-zlib) [ 2599.805991] Lustre: Lustre: Build Version: 2.16.56_1_g45307c0 [ 2599.859979] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2599.861707] LNet: Accept secure, port 988 [ 2601.388680] Key type lgssc registered [ 2601.626249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2603.817813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2605.541975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2607.053629] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2607.056342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2608.641530] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2608.645480] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2610.332730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2610.346448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2610.351346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.419270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2611.431438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2611.465381] Lustre: lustre-MDT0000: new disk, initializing [ 2611.483687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.489711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2612.288159] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2615.755892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2615.776041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2615.801024] Lustre: 27940:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2615.818449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2615.820491] Lustre: Skipped 1 previous similar message [ 2615.853643] Lustre: lustre-MDT0001: new disk, initializing [ 2615.874661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2615.885452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2615.888252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2616.682427] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2619.201363] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2620.042986] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2621.790821] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2621.794869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2621.814464] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2621.818225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2621.897492] Lustre: lustre-OST0000: new disk, initializing [ 2621.900112] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2621.919030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2623.173761] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2625.626771] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2629.704897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2629.707967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2629.726166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2633.684229] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 2640.738639] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 2647.333485] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 2652.698250] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 2689.819075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2689.819678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2689.819998] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2689.830596] LustreError: Skipped 1 previous similar message [ 2691.713066] LustreError: 32656:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2691.757730] Lustre: server umount lustre-OST0000 complete [ 2694.827233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.827491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2694.827493] Lustre: Skipped 1 previous similar message [ 2694.834622] Lustre: Skipped 2 previous similar messages [ 2698.732862] LustreError: 460:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2698.735330] LustreError: 460:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2698.751494] Lustre: server umount lustre-MDT0000 complete [ 2699.726231] LustreError: 27931:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749608093 with bad export cookie 4629951565600675577 [ 2699.727184] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2699.734190] LustreError: 27931:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2699.777763] LustreError: 739:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2699.779643] LustreError: 739:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2699.848116] Lustre: server umount lustre-MDT0001 complete [ 2702.068011] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2702.885860] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2704.153985] LustreError: 1946:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2704.155994] LustreError: 1946:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2704.170806] Lustre: server umount MGS complete [ 2707.051791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2707.184635] LustreError: 2773:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 2707.204488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2707.987660] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2710.282391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2711.213034] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2712.480900] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2713.275365] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2715.013370] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2715.017933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2715.110724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2715.113186] Lustre: Skipped 1 previous similar message [ 2716.324378] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2717.521103] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2735.131445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.131769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2735.131770] Lustre: Skipped 1 previous similar message [ 2735.137719] Lustre: Skipped 1 previous similar message [ 2738.153605] LustreError: 6086:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2738.155848] LustreError: 6086:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2738.205995] Lustre: server umount lustre-OST0000 complete [ 2740.139087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2740.143885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2740.146006] Lustre: Skipped 2 previous similar messages [ 2745.147109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2745.149340] Lustre: Skipped 1 previous similar message [ 2745.234047] Lustre: server umount lustre-MDT0000 complete [ 2746.211527] LustreError: 5730:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749608140 with bad export cookie 4629951565600677404 [ 2746.212939] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2746.219073] LustreError: 5730:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2746.271951] LustreError: 6627:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2746.274019] LustreError: 6627:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2746.342929] Lustre: server umount lustre-MDT0001 complete [ 2749.477988] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2750.087080] Key type lgssc unregistered [ 2750.174926] LNet: 7151:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2751.180855] LNet: Removed LNI 192.168.203.144@tcp [ 2756.035310] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 2760.330940] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2760.504810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2760.527060] alg: No test for adler32 (adler32-zlib) [ 2761.350749] Lustre: Lustre: Build Version: 2.16.56_1_g45307c0 [ 2761.406292] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2761.409870] LNet: Accept secure, port 988 [ 2762.942676] Key type lgssc registered [ 2763.169699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2765.312651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2766.801003] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2766.804649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2768.433324] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2770.378442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2772.123771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2772.143039] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2772.148337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2773.222991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2773.234137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2773.270555] Lustre: lustre-MDT0000: new disk, initializing [ 2773.288713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2773.295486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2774.088350] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2777.596818] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2777.600354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2777.616177] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2777.619265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2777.698648] Lustre: lustre-OST0000: new disk, initializing [ 2777.701139] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2777.703069] Lustre: Skipped 1 previous similar message [ 2777.719046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2778.203388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2778.207646] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2778.217188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2779.004668] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2923.435298] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2923.440010] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2928.145352] LustreError: 12996:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 2928.262123] Lustre: server umount lustre-MDT0000 complete [ 2929.185274] LustreError: 10812:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749608323 with bad export cookie 2135312731487504164 [ 2929.186304] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2929.191191] LustreError: 10812:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2929.209362] LustreError: 13269:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2929.211188] LustreError: 13269:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2929.255170] Lustre: server umount lustre-OST0000 complete [ 2935.135108] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 2937.119543] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2939.568802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2941.316805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2942.832392] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2942.836132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2944.388483] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2944.393445] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2946.159841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2946.175606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2946.241309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2946.249918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2946.283869] Lustre: lustre-MDT0000: new disk, initializing [ 2946.301291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.306878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2947.094198] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2950.599093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2950.622438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2950.646960] Lustre: 17056:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2950.665136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2950.667135] Lustre: Skipped 1 previous similar message [ 2950.703852] Lustre: lustre-MDT0001: new disk, initializing [ 2950.728558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2950.739887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2950.743202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2951.528151] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2954.083391] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2954.911935] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2956.728607] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2956.733133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2956.753553] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2956.757782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2956.850218] Lustre: lustre-OST0000: new disk, initializing [ 2956.852665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2956.872788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2958.171169] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2960.681217] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2961.489558] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2963.842142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2963.850080] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2963.863943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2964.588269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2965.384811] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2965.441755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2968.859498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2968.859820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2968.864886] Lustre: Skipped 1 previous similar message [ 2971.952817] LustreError: 20612:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2971.955021] LustreError: 20612:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2972.026746] Lustre: server umount lustre-OST0000 complete [ 2973.867339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2973.873214] Lustre: Skipped 1 previous similar message [ 2973.873568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2973.873570] Lustre: Skipped 1 previous similar message [ 2978.875183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2978.877010] Lustre: Skipped 2 previous similar messages [ 2979.050077] LustreError: 20883:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2979.051766] LustreError: 20883:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2979.067768] Lustre: server umount lustre-MDT0000 complete [ 2980.009467] LustreError: 18008:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749608374 with bad export cookie 2135312731490444948 [ 2980.010306] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2980.017013] LustreError: 18008:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2980.117797] Lustre: server umount lustre-MDT0001 complete [ 2980.961870] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 2981.331270] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 22:19:35 (1749608375) [ 2983.059986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.205999] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2983.921985] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2986.217757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2987.142547] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2988.445959] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2989.282967] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2991.037791] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2991.040818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2992.436596] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2993.743411] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2995.532671] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2995.536656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2995.556362] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2995.559452] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2995.601155] Lustre: lustre-OST0001: new disk, initializing [ 2995.603316] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2996.927918] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2999.489062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2999.493085] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2999.502174] LustreError: 25770:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 2999.506306] LustreError: 25770:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 2999.769224] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2999.817115] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3000.504636] LustreError: 25776:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 3000.626211] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b6fc7000.ost_server_uuid 50 [ 3001.006192] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b6fc7000.ost_server_uuid in FULL state after 0 sec [ 3001.509736] LustreError: 25776:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 3001.513292] LustreError: 25776:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 3001.884485] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6fc7000.ost_server_uuid 50 [ 3002.261653] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8800b6fc7000.ost_server_uuid in FULL state after 0 sec [ 3003.524838] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3013.531113] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3013.531207] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3013.531563] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3013.539141] LustreError: Skipped 1 previous similar message [ 3019.426567] LustreError: 26583:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3019.429001] LustreError: 26583:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3019.440121] Lustre: server umount lustre-OST0001 complete [ 3019.531753] LustreError: 25673:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3019.536865] LustreError: 25673:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3021.675192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3021.677824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.681179] Lustre: Skipped 1 previous similar message [ 3021.683132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3021.684884] Lustre: Skipped 5 previous similar messages [ 3023.179174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3024.539652] LustreError: 24437:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.543664] LustreError: 24437:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3026.683482] Lustre: server umount lustre-OST0000 complete [ 3033.754949] LustreError: 27123:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3033.757280] LustreError: 27123:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3033.773435] Lustre: server umount lustre-MDT0000 complete [ 3034.554979] LustreError: 21902:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3034.560225] LustreError: 21902:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3034.788464] LustreError: 21883:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749608429 with bad export cookie 2135312731490445816 [ 3034.789802] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3034.796297] LustreError: 21883:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3034.932796] Lustre: server umount lustre-MDT0001 complete [ 3037.991700] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 3038.546546] Key type lgssc unregistered [ 3038.630803] LNet: 27916:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3039.635843] LNet: Removed LNI 192.168.203.144@tcp [ 3046.290019] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3046.467666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3046.471972] alg: No test for adler32 (adler32-zlib) [ 3047.284570] Lustre: Lustre: Build Version: 2.16.56_1_g45307c0 [ 3047.338758] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 3047.340593] LNet: Accept secure, port 988 [ 3048.870686] Key type lgssc registered [ 3049.070410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3050.926066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3050.931109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.056716] LustreError: 29438:0:(ldlm_lib.c:523:client_obd_setup()) can't add initial connection [ 3052.060683] LustreError: 29438:0:(osp_dev.c:1207:osp_init0()) lustre-MDT0001-osp-MDT0000: can't setup obd: rc = -2 [ 3052.065445] LustreError: 29438:0:(obd_config.c:779:class_setup()) setup lustre-MDT0001-osp-MDT0000 failed (-2) [ 3052.069633] LustreError: 29438:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -2 [ 3052.073348] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.144@tcp [ 3052.076075] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3052.080377] LustreError: 29401:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3052.083435] LustreError: 29401:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -2 [ 3052.086352] Lustre: Failing over lustre-MDT0000 [ 3052.142313] LustreError: 29401:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3052.207745] Lustre: server umount lustre-MDT0000 complete [ 3052.211683] LustreError: 29401:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3052.595717] Lustre: DEBUG MARKER: conf-sanity test_112b: @@@@@@ FAIL: MDS start failed [ 3053.928797] Lustre: DEBUG MARKER: == conf-sanity test 113: Shadow mountpoint correctly report ro/rw for mounts ========================================================== 22:20:47 (1749608447) [ 3063.134434] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3065.551492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3065.668177] LustreError: 31838:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 3065.672261] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3065.675620] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3065.680639] LustreError: 31802:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 3065.683662] LustreError: 31802:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -17 [ 3065.686657] Lustre: Failing over lustre-MDT0000 [ 3065.741307] LustreError: 31802:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3065.743452] LustreError: 31802:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3065.794744] Lustre: server umount lustre-MDT0000 complete [ 3065.795929] LustreError: 31802:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 3067.584317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3084.673809] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3090.675688] LustreError: 32299:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.144@tcp: failed processing log, type 4: rc = -110 [ 3108.727756] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3108.730736] Lustre: 32299:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3112.728066] LustreError: 32318:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3112.732387] LustreError: 32318:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3117.738860] LustreError: 32318:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3119.915797] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3119.919085] Lustre: 32311:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3120.739626] LustreError: 32314:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3120.746768] LustreError: 32314:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3125.754935] LustreError: 32318:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3125.759490] LustreError: 32318:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3131.739592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3132.536480] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3132.763013] LustreError: 32318:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3132.768047] LustreError: 32318:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3133.291012] LustreError: 369:0:(obd_class.h:1230:obd_iocontrol()) MGC192.168.203.144@tcp: no obd_iocontrol operation [ 3135.006563] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3135.010267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3142.779062] LustreError: 32318:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3142.783354] LustreError: 32318:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3147.923963] LustreError: 32311:0:(mgc_request.c:614:do_requeue()) failed processing log: -110 [ 3153.927949] LustreError: 32311:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3159.929910] LustreError: 32311:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3161.796314] LustreError: 32318:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3161.799609] LustreError: 32318:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 10 previous similar messages [ 3181.082727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3182.379960] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3183.316869] LustreError: 32311:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3184.708822] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3184.712770] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3192.107401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 3197.115146] LustreError: 32704:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3197.119692] LustreError: 32704:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 22 previous similar messages [ 3199.318891] Lustre: 32311:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3205.321876] LustreError: 32311:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3207.131365] LustreError: lustre-MDT0000-lwp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 3217.323844] LustreError: 32311:0:(mgc_request.c:614:do_requeue()) failed processing log: -110 [ 3230.747740] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3231.936322] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3251.226999] LustreError: 32311:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3251.229962] LustreError: 32311:0:(mgc_request.c:614:do_requeue()) Skipped 2 previous similar messages [ 3256.811812] LustreError: lustre-MDT0000-lwp-OST0000: operation mds_connect to node 0@lo failed: rc = -19 [ 3261.814821] LustreError: 32314:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3261.819371] LustreError: 32314:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 51 previous similar messages [ 3283.232221] LustreError: 32311:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3283.238910] LustreError: 32311:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 3311.894358] LustreError: lustre-MDT0000-lwp-OST0001: operation mds_connect to node 0@lo failed: rc = -19 [ 3336.932544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 3347.244879] LustreError: 32311:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3347.246979] LustreError: 32311:0:(mgc_request.c:614:do_requeue()) Skipped 3 previous similar messages [ 3351.963572] LustreError: lustre-MDT0000-lwp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 3363.250887] Lustre: 32311:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3392.027020] LustreError: 32318:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.031542] LustreError: 32318:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 103 previous similar messages [ 3407.051719] LustreError: lustre-MDT0000-lwp-OST0000: operation mds_connect to node 0@lo failed: rc = -19 [ 3462.132851] LustreError: lustre-MDT0000-lwp-OST0001: operation mds_connect to node 0@lo failed: rc = -19 [ 3489.644913] LustreError: 32311:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3489.647303] LustreError: 32311:0:(mgc_request.c:614:do_requeue()) Skipped 6 previous similar messages [ 3557.291870] LustreError: lustre-MDT0000-lwp-OST0000: operation mds_connect to node 0@lo failed: rc = -19 [ 3557.294208] LustreError: Skipped 2 previous similar messages [ 3558.424171] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3559.318668] LustreError: 3030:0:(obd_class.h:1230:obd_iocontrol()) MGC192.168.203.144@tcp: no obd_iocontrol operation [ 3569.303704] LustreError: 32363:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88009c884000 x1834597420743680/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3569.649960] Lustre: 32311:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3569.650417] LustreError: 32398:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000009:0x0:0x0]: rc = -5 [ 3569.650476] LustreError: 32398:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't get id from catalogs: rc = -5 [ 3569.650482] LustreError: 32398:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0000-osp-MDT0001: get update log duration 438, retries 0, failed: rc = -5 [ 3569.772098] LustreError: 3755:0:(obd_class.h:479:obd_check_dev()) Device 7 not setup [ 3569.774684] LustreError: 3755:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3569.835804] Lustre: server umount lustre-MDT0001 complete [ 3581.331610] LustreError: 4024:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3581.337836] LustreError: 4024:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3581.351977] Lustre: server umount lustre-OST0000 complete [ 3592.413970] LustreError: 4298:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3592.416938] LustreError: 4298:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3592.470992] Lustre: server umount lustre-OST0001 complete [ 3596.007243] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3598.360099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3598.475223] LustreError: 5722:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 3598.478214] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3598.481411] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3598.485425] LustreError: 5686:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 3598.487836] LustreError: 5686:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -17 [ 3598.489966] Lustre: Failing over lustre-MDT0000 [ 3598.584246] Lustre: server umount lustre-MDT0000 complete [ 3598.585745] LustreError: 5686:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 3598.958721] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 3599.353825] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 3599.734621] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 22:29:53 (1749608993) [ 3604.434404] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3606.458866] SGI XFS with ACLs, security attributes, no debug enabled [ 3606.463766] XFS (loop0): Mounting V5 Filesystem [ 3606.470454] XFS (loop0): Ending clean mount [ 3608.720579] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5 [ 3608.848071] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3612.819306] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 3614.574949] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3616.778273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3618.394636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3619.887306] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3619.890797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3621.371326] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3621.374871] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3623.121116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3623.139567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3623.236189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3623.247942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3623.294496] Lustre: lustre-MDT0000: new disk, initializing [ 3623.313863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3623.321217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3624.268994] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3628.313984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3628.341732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3628.377107] Lustre: 10969:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3628.405656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3628.409071] Lustre: Skipped 1 previous similar message [ 3628.463207] Lustre: lustre-MDT0001: new disk, initializing [ 3628.496088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3628.511411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3628.516113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3629.426387] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3629.467698] LustreError: 12209:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 3629.471427] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3631.826676] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3632.773154] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3634.355276] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3634.358356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3634.374304] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3634.377192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3634.440366] Lustre: lustre-OST0000: new disk, initializing [ 3634.442064] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3635.593058] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3636.277939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3636.280724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3636.295986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3638.005097] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3638.836550] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3638.879861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3639.967512] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3640.053090] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3641.291320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3641.291801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3641.296056] Lustre: Skipped 1 previous similar message [ 3646.299221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3646.301250] Lustre: Skipped 2 previous similar messages [ 3646.646580] LustreError: 14510:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3646.648122] LustreError: 14510:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3646.706601] Lustre: server umount lustre-OST0000 complete [ 3648.507280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3648.511663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3651.307215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3651.307459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3651.312570] Lustre: Skipped 1 previous similar message [ 3653.724895] Lustre: server umount lustre-MDT0000 complete [ 3654.754656] LustreError: 13128:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749609048 with bad export cookie 14823519052493312343 [ 3654.756482] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3654.763652] LustreError: 13128:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3654.884818] Lustre: server umount lustre-MDT0001 complete [ 3656.709061] XFS (loop0): Unmounting Filesystem [ 3657.908625] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 22:30:51 (1749609051) [ 3659.621643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3659.746178] LustreError: 16121:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3659.754640] LustreError: 16121:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 140 previous similar messages [ 3659.760602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3659.762332] Lustre: Skipped 1 previous similar message [ 3660.483720] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3662.567842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3663.400057] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3664.620676] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3665.445367] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3667.111585] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3667.114471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3668.458729] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3669.691798] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3677.211640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3677.212177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3677.220054] Lustre: Skipped 1 previous similar message [ 3678.922399] LustreError: 19018:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3678.924657] LustreError: 19018:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3678.972936] Lustre: server umount lustre-OST0000 complete [ 3682.219233] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3682.223607] Lustre: Skipped 1 previous similar message [ 3686.055993] Lustre: server umount lustre-MDT0000 complete [ 3687.066818] LustreError: 16103:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749609081 with bad export cookie 14823519052493313225 [ 3687.069009] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3687.077333] LustreError: 16103:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3687.205947] Lustre: server umount lustre-MDT0001 complete [ 3690.417547] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 3691.226379] Key type lgssc unregistered [ 3691.334934] LNet: 20081:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3692.343820] LNet: Removed LNI 192.168.203.144@tcp [ 3699.542561] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3699.718021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3699.722326] alg: No test for adler32 (adler32-zlib) [ 3700.546841] Lustre: Lustre: Build Version: 2.16.56_1_g45307c0 [ 3700.612654] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 3700.617800] LNet: Accept secure, port 988 [ 3702.154666] Key type lgssc registered [ 3702.364760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3704.216606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3704.221382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3705.333063] LustreError: 21799:0:(ldlm_lib.c:523:client_obd_setup()) can't add initial connection [ 3705.335653] LustreError: 21799:0:(osp_dev.c:1207:osp_init0()) lustre-MDT0001-osp-MDT0000: can't setup obd: rc = -2 [ 3705.338634] LustreError: 21799:0:(obd_config.c:779:class_setup()) setup lustre-MDT0001-osp-MDT0000 failed (-2) [ 3705.341475] LustreError: 21799:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -2 [ 3705.344596] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.144@tcp [ 3705.347377] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3705.352125] LustreError: 21762:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3705.354490] LustreError: 21762:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -2 [ 3705.357025] Lustre: Failing over lustre-MDT0000 [ 3705.407000] LustreError: 21762:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3705.471960] Lustre: server umount lustre-MDT0000 complete [ 3705.473078] LustreError: 21762:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3705.808320] Lustre: DEBUG MARKER: conf-sanity test_119: @@@@@@ FAIL: MDT start failed [ 3706.972895] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 22:31:40 (1749609100) [ 3708.709205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3708.840685] LustreError: 22622:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 3708.843393] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3708.845843] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3708.851274] LustreError: 22584:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 3708.854250] LustreError: 22584:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -17 [ 3708.857082] Lustre: Failing over lustre-MDT0000 [ 3708.913238] LustreError: 22584:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3708.915050] LustreError: 22584:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3708.977920] Lustre: server umount lustre-MDT0000 complete [ 3708.979077] LustreError: 22584:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 3709.389551] Lustre: DEBUG MARKER: conf-sanity test_120: @@@@@@ FAIL: MDT start failed [ 3710.694277] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:31:44 (1749609104) [ 3715.175669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3715.296816] LustreError: 24247:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 3715.300008] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3715.303259] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3715.307628] LustreError: 24210:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 3715.310169] LustreError: 24210:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -17 [ 3715.312684] Lustre: Failing over lustre-MDT0000 [ 3715.364103] LustreError: 24210:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3715.366069] LustreError: 24210:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3715.414668] Lustre: server umount lustre-MDT0000 complete [ 3715.416144] LustreError: 24210:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 3715.773953] Lustre: DEBUG MARKER: conf-sanity test_121: @@@@@@ FAIL: MGS MDS Start failed [ 3717.015181] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:31:50 (1749609110) [ 3720.731454] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 3722.472548] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3724.609980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3726.308427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3727.834604] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3727.838337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3729.450439] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3729.453846] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3731.261930] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3733.819930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3733.837139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3733.901671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3733.912531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3733.946181] Lustre: lustre-MDT0000: new disk, initializing [ 3733.964866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3733.970272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3734.731456] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3738.173963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3738.192034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3738.218371] Lustre: 28390:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3738.236593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3738.239048] Lustre: Skipped 1 previous similar message [ 3738.278362] Lustre: lustre-MDT0001: new disk, initializing [ 3738.300836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3738.312954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3738.315977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3739.072148] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3741.552166] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3742.391402] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3743.138475] LustreError: 30237:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 3743.141188] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3743.143643] LustreError: 28392:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3744.082346] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3744.086433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3744.106127] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3744.108799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3744.193950] Lustre: lustre-OST0000: new disk, initializing [ 3744.195825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3744.215533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3745.441909] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3747.912235] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3750.638597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3750.641658] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3750.650958] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 3750.662954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3750.668735] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 3755.339024] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 3760.667319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3760.667441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3760.667753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3763.917551] LustreError: 31662:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3763.920346] LustreError: 31662:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3763.978709] Lustre: server umount lustre-OST0000 complete [ 3765.675267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3765.675584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3765.675585] Lustre: Skipped 1 previous similar message [ 3765.681238] Lustre: Skipped 2 previous similar messages [ 3770.683083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3770.686016] Lustre: Skipped 2 previous similar messages [ 3770.973681] LustreError: 31931:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3770.976323] LustreError: 31931:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3770.991831] Lustre: server umount lustre-MDT0000 complete [ 3771.950242] LustreError: 28382:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1749609166 with bad export cookie 454833608235431900 [ 3771.951183] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3771.956955] LustreError: 28382:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3772.089018] Lustre: server umount lustre-MDT0001 complete [ 3775.092604] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 3775.656317] Key type lgssc unregistered [ 3775.741932] LNet: 32726:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3776.746855] LNet: Removed LNI 192.168.203.144@tcp [ 3783.269361] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3783.392894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3783.420545] alg: No test for adler32 (adler32-zlib) [ 3784.232933] Lustre: Lustre: Build Version: 2.16.56_1_g45307c0 [ 3784.279620] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 3784.281358] LNet: Accept secure, port 988 [ 3785.811694] Key type lgssc registered [ 3786.018305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3787.890478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3787.897034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3789.012732] LustreError: 1820:0:(ldlm_lib.c:523:client_obd_setup()) can't add initial connection [ 3789.015045] LustreError: 1820:0:(osp_dev.c:1207:osp_init0()) lustre-MDT0001-osp-MDT0000: can't setup obd: rc = -2 [ 3789.017629] LustreError: 1820:0:(obd_config.c:779:class_setup()) setup lustre-MDT0001-osp-MDT0000 failed (-2) [ 3789.020146] LustreError: 1820:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -2 [ 3789.022989] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.144@tcp [ 3789.026010] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3789.030216] LustreError: 1783:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3789.032592] LustreError: 1783:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -2 [ 3789.035018] Lustre: Failing over lustre-MDT0000 [ 3789.083102] LustreError: 1783:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3789.129621] Lustre: server umount lustre-MDT0000 complete [ 3789.130981] LustreError: 1783:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3789.484018] Lustre: DEBUG MARKER: conf-sanity test_123aa: @@@@@@ FAIL: MDT start failed [ 3790.643309] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:33:04 (1749609184) [ 3792.283989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3792.396677] LustreError: 2644:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 3792.401407] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3792.403869] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3792.409181] LustreError: 2607:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 3792.412037] LustreError: 2607:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -17 [ 3792.414858] Lustre: Failing over lustre-MDT0000 [ 3792.466098] LustreError: 2607:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3792.467963] LustreError: 2607:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3792.514641] Lustre: server umount lustre-MDT0000 complete [ 3792.515914] LustreError: 2607:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 3792.856295] Lustre: DEBUG MARKER: conf-sanity test_123ab: @@@@@@ FAIL: MDT start failed [ 3794.029481] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:33:07 (1749609187) [ 3795.971166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3796.088299] LustreError: 3532:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 3796.091063] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3796.093915] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3796.100069] LustreError: 3495:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 3796.102633] LustreError: 3495:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -17 [ 3796.106858] Lustre: Failing over lustre-MDT0000 [ 3796.158162] LustreError: 3495:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3796.159908] LustreError: 3495:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3796.208995] Lustre: server umount lustre-MDT0000 complete [ 3796.210337] LustreError: 3495:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 3796.565500] Lustre: DEBUG MARKER: conf-sanity test_123ac: @@@@@@ FAIL: MDT start failed [ 3797.814936] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:33:11 (1749609191) [ 3799.553662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3799.673280] LustreError: 4358:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 3799.676328] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3799.679219] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3799.684198] LustreError: 4321:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 3799.686819] LustreError: 4321:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -17 [ 3799.689358] Lustre: Failing over lustre-MDT0000 [ 3799.738279] LustreError: 4321:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3799.740532] LustreError: 4321:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3799.791674] Lustre: server umount lustre-MDT0000 complete [ 3799.793525] LustreError: 4321:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 3800.153579] Lustre: DEBUG MARKER: conf-sanity test_123ad: @@@@@@ FAIL: MDT start failed [ 3801.326385] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:33:15 (1749609195) [ 3804.381813] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3806.724253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3806.844977] LustreError: 5966:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 3806.848062] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3806.852107] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3806.859827] LustreError: 5930:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 3806.864134] LustreError: 5930:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -17 [ 3806.867985] Lustre: Failing over lustre-MDT0000 [ 3806.920294] LustreError: 5930:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3806.923367] LustreError: 5930:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3806.976305] Lustre: server umount lustre-MDT0000 complete [ 3806.978470] LustreError: 5930:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 3808.713134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3825.780849] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3831.782741] LustreError: 6427:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.144@tcp: failed processing log, type 4: rc = -110 [ 3849.830777] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3849.832734] Lustre: 6427:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3852.831045] LustreError: 6490:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3852.838739] LustreError: 6490:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3857.834865] LustreError: 6490:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3861.367798] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3861.370453] Lustre: 6439:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3861.845270] LustreError: 6442:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3861.851202] LustreError: 6442:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3866.858938] LustreError: 6446:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3866.864751] LustreError: 6446:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3871.866886] LustreError: 6446:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3871.871048] LustreError: 6446:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3872.844666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3873.561107] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3874.278639] LustreError: 6963:0:(obd_class.h:1230:obd_iocontrol()) MGC192.168.203.144@tcp: no obd_iocontrol operation [ 3875.950815] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3875.954799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3882.891128] LustreError: 6446:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3882.895830] LustreError: 6446:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 3889.375971] LustreError: 6439:0:(mgc_request.c:614:do_requeue()) failed processing log: -110 [ 3895.378992] LustreError: 6439:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3901.381058] LustreError: 6439:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3902.916386] LustreError: 6446:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3902.922243] LustreError: 6446:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 7 previous similar messages [ 3922.038784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3922.939959] LustreError: 6439:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3923.230010] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3925.390793] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3925.394259] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3925.406235] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3925.409895] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3933.067349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 3938.075043] LustreError: 6490:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3938.080114] LustreError: 6490:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 22 previous similar messages [ 3938.942932] LustreError: 6439:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3944.944910] Lustre: 6439:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3948.091258] LustreError: lustre-MDT0000-lwp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 3950.948982] LustreError: 6439:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3956.424734] LustreError: lustre-OST0001: cannot register this server with the MGS: rc = -110. Is the MGS running? [ 3956.428559] LustreError: 8455:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -110 [ 3956.431903] LustreError: 8455:0:(tgt_mount.c:1762:server_put_super()) no obd lustre-OST0001 [ 3956.435629] LustreError: 8455:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0001 not registered [ 3956.440627] Lustre: server umount lustre-OST0001 complete [ 3956.442475] LustreError: 8455:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -110 [ 3972.951828] LustreError: 6439:0:(mgc_request.c:614:do_requeue()) failed processing log: -110 [ 4003.171462] LustreError: 6490:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4003.171932] LustreError: lustre-MDT0000-lwp-OST0000: operation mds_connect to node 0@lo failed: rc = -19 [ 4003.177176] LustreError: 6490:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 38 previous similar messages [ 4016.293952] LustreError: 6439:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 4016.296238] LustreError: 6439:0:(mgc_request.c:614:do_requeue()) Skipped 2 previous similar messages [ 4080.298844] Lustre: 6439:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4083.307304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 4096.300908] LustreError: 6439:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 4096.303254] LustreError: 6439:0:(mgc_request.c:614:do_requeue()) Skipped 3 previous similar messages [ 4098.331457] LustreError: lustre-MDT0000-lwp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 4133.386972] LustreError: 6446:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4133.394562] LustreError: 6446:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 77 previous similar messages [ 4153.412689] LustreError: lustre-MDT0000-lwp-OST0000: operation mds_connect to node 0@lo failed: rc = -19 [ 4233.547385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 4237.180924] Lustre: 6439:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4253.183988] LustreError: 6439:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 4253.187142] LustreError: 6439:0:(mgc_request.c:614:do_requeue()) Skipped 7 previous similar messages [ 4282.855598] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4283.713324] LustreError: 9201:0:(obd_class.h:1230:obd_iocontrol()) MGC192.168.203.144@tcp: no obd_iocontrol operation [ 4289.202870] LustreError: 9335:0:(obd_class.h:1230:obd_iocontrol()) MGC192.168.203.144@tcp: no obd_iocontrol operation [ 4290.234963] Lustre: DEBUG MARKER: conf-sanity test_123ae: @@@@@@ FAIL: new set_param -P wasn't stored in params log [ 4291.638712] LustreError: 9833:0:(obd_class.h:1230:obd_iocontrol()) MGC192.168.203.144@tcp: no obd_iocontrol operation [ 4291.641669] LustreError: 9833:0:(obd_class.h:1230:obd_iocontrol()) Skipped 3 previous similar messages [ 4292.166741] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:41:25 (1749609685) [ 4295.309660] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 4297.599001] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4297.680693] LustreError: 11160:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 4297.685762] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 4297.689045] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4297.693024] LustreError: 11150:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 4297.695202] LustreError: 11150:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -17 [ 4297.697199] Lustre: Failing over lustre-MDT0000 [ 4297.756019] Lustre: server umount lustre-MDT0000 complete [ 4297.758031] LustreError: 11150:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 4299.751222] LustreError: 11686:0:(obd_class.h:1230:obd_iocontrol()) MGC192.168.203.144@tcp: no obd_iocontrol operation [ 4302.635098] LustreError: lustre-MDT0000-lwp-OST0000: operation mds_connect to node 0@lo failed: rc = -19 [ 4302.637111] LustreError: Skipped 1 previous similar message [ 4303.600043] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4303.602747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4314.653601] Lustre: 6439:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1749609692/real 1749609692] req@ffff880130776680 x1834598193493376/t0(0) o101->MGC192.168.203.144@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1749609708 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4314.662147] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4322.664767] LustreError: lustre-OST0001: cannot register this server with the MGS: rc = -108. Is the MGS running? [ 4322.667194] LustreError: 12732:0:(tgt_mount.c:1762:server_put_super()) no obd lustre-OST0001 [ 4322.669111] LustreError: 12732:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0001 not registered [ 4322.672718] Lustre: server umount lustre-OST0001 complete [ 4379.771125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 4393.786981] LustreError: 6446:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4393.793935] LustreError: 6446:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 158 previous similar messages [ 4394.664765] Lustre: 6439:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4516.544720] LustreError: 6439:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 4516.546687] LustreError: 6439:0:(mgc_request.c:614:do_requeue()) Skipped 10 previous similar messages [ 4529.002975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 4529.005372] LustreError: Skipped 2 previous similar messages [ 4576.549768] Lustre: 6439:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4649.081067] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4649.870460] LustreError: 13468:0:(obd_class.h:1230:obd_iocontrol()) MGC192.168.203.144@tcp: no obd_iocontrol operation [ 4655.908603] Lustre: DEBUG MARKER: conf-sanity test_123af: @@@@@@ FAIL: 0 != 0 - 1 [ 4657.417231] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:47:31 (1749610051) [ 4659.104628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4659.203438] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0x0 to 0xae754a0887514aa0 [ 4659.205854] Lustre: MGC192.168.203.144@tcp: Connection restored to (at 0@lo) [ 4659.220571] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 4659.246374] LustreError: 14527:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 4659.248834] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 4659.251116] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4659.258685] LustreError: 14513:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 4659.262463] LustreError: 14513:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -17 [ 4659.264691] LustreError: 14513:0:(tgt_mount.c:2268:server_fill_super()) Skipped 1 previous similar message [ 4659.268326] Lustre: Failing over lustre-MDT0000 [ 4659.333312] Lustre: server umount lustre-MDT0000 complete [ 4659.335330] LustreError: 14513:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 4659.338516] LustreError: 14513:0:(super25.c:181:lustre_fill_super()) Skipped 1 previous similar message [ 4659.687585] Lustre: DEBUG MARKER: conf-sanity test_123ag: @@@@@@ FAIL: MDT start failed [ 4660.874751] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:47:34 (1749610054) [ 4663.854812] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 4666.028917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4666.054860] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4666.099274] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 4666.101982] Lustre: Failing over lustre-MDT0000 [ 4666.154456] Lustre: server umount lustre-MDT0000 complete [ 4668.104818] LustreError: 16625:0:(obd_class.h:1230:obd_iocontrol()) MGC192.168.203.144@tcp: no obd_iocontrol operation [ 4668.108883] LustreError: 16625:0:(obd_class.h:1230:obd_iocontrol()) Skipped 2 previous similar messages [ 4671.985711] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4671.988699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4683.361692] Lustre: 6439:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1749610061/real 1749610061] req@ffff8800974c1f80 x1834598193543936/t0(0) o101->MGC192.168.203.144@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1749610077 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4683.368334] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4691.372724] LustreError: lustre-OST0001: cannot register this server with the MGS: rc = -108. Is the MGS running? [ 4691.377047] LustreError: 17670:0:(tgt_mount.c:1762:server_put_super()) no obd lustre-OST0001 [ 4691.380278] LustreError: 17670:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0001 not registered [ 4691.385570] Lustre: server umount lustre-OST0001 complete [ 4763.373737] Lustre: 6439:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4828.595263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 4828.598703] LustreError: Skipped 5 previous similar messages [ 4908.730812] LustreError: 6446:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4908.735282] LustreError: 6446:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 308 previous similar messages [ 4964.700656] Lustre: 6439:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 5017.765027] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5018.573299] LustreError: 18408:0:(obd_class.h:1230:obd_iocontrol()) MGC192.168.203.144@tcp: no obd_iocontrol operation [ 5024.415866] Lustre: DEBUG MARKER: conf-sanity test_123ah: @@@@@@ FAIL: lustre-OST0000 not found (client) [ 5025.941664] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:53:39 (1749610419) [ 5027.637338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5027.907204] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0x0 to 0xae754a0887514f54 [ 5027.909669] Lustre: MGC192.168.203.144@tcp: Connection restored to 192.168.203.144@tcp (at 0@lo) [ 5027.949378] LustreError: 19404:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 5027.952098] LustreError: 19404:0:(obd_config.c:1993:class_config_llog_handler()) Skipped 1 previous similar message [ 5027.954125] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 5027.956483] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5027.960695] LustreError: Skipped 1 previous similar message [ 5027.961855] LustreError: 19391:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 5027.964352] LustreError: 19391:0:(tgt_mount.c:1523:server_start_targets()) Skipped 1 previous similar message [ 5027.966492] LustreError: 19391:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -17 [ 5027.968403] LustreError: 19391:0:(tgt_mount.c:2268:server_fill_super()) Skipped 2 previous similar messages [ 5027.970802] Lustre: Failing over lustre-MDT0000 [ 5028.020226] Lustre: server umount lustre-MDT0000 complete [ 5028.021474] LustreError: 19391:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 5028.023390] LustreError: 19391:0:(super25.c:181:lustre_fill_super()) Skipped 2 previous similar messages [ 5028.377040] Lustre: DEBUG MARKER: conf-sanity test_123ai: @@@@@@ FAIL: MDT start failed [ 5029.508344] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:53:43 (1749610423) [ 5031.121198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5031.143129] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5031.145908] LustreError: 6439:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 5031.147726] LustreError: 6439:0:(mgc_request.c:614:do_requeue()) Skipped 19 previous similar messages [ 5031.188343] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 5031.192036] Lustre: Failing over lustre-MDT0000 [ 5031.239973] Lustre: server umount lustre-MDT0000 complete [ 5031.589461] Lustre: DEBUG MARKER: conf-sanity test_123aj: @@@@@@ FAIL: MDT start failed [ 5032.701493] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:53:46 (1749610426) [ 5034.309885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5034.370539] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 5034.372990] Lustre: Failing over lustre-MDT0000 [ 5034.423202] Lustre: server umount lustre-MDT0000 complete [ 5034.765530] Lustre: DEBUG MARKER: conf-sanity test_123F: @@@@@@ FAIL: MDT start failed [ 5035.903630] Lustre: DEBUG MARKER: == conf-sanity test 123G: clear and reset all parameters using apply_yaml ========================================================== 22:53:49 (1749610429) [ 5037.578785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5037.602006] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5037.604542] LustreError: Skipped 1 previous similar message [ 5037.644947] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 5037.647694] Lustre: Failing over lustre-MDT0000 [ 5037.706299] Lustre: server umount lustre-MDT0000 complete [ 5038.072796] Lustre: DEBUG MARKER: conf-sanity test_123G: @@@@@@ FAIL: MDT start failed [ 5039.188738] Lustre: DEBUG MARKER: == conf-sanity test 123H: check concurent accesses with 'lctl llog_print ========================================================== 22:53:52 (1749610432) [ 5040.843267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5040.906570] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 5040.910076] Lustre: Failing over lustre-MDT0000 [ 5040.964717] Lustre: server umount lustre-MDT0000 complete [ 5041.321372] Lustre: DEBUG MARKER: conf-sanity test_123H: @@@@@@ FAIL: MDT start failed [ 5042.473522] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:53:56 (1749610436) [ 5042.807959] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 5043.194976] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:53:56 (1749610436) [ 5058.581628] Lustre: 6439:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1749610436/real 1749610436] req@ffff8800966f0380 x1834598193639424/t0(0) o101->MGC192.168.203.144@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1749610452 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5058.588947] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5058.591480] LustreError: Skipped 1 previous similar message [ 5063.595003] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5063.598381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5064.610210] LustreError: 23194:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 5064.612004] LustreError: 23194:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5064.662450] Lustre: server umount lustre-OST0000 complete [ 5066.226554] LustreError: 6523:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000009:0x0:0x0]: rc = -5 [ 5066.230940] LustreError: 6523:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't get id from catalogs: rc = -5 [ 5066.233856] LustreError: 6523:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0000-osp-MDT0001: get update log duration 1193, retries 0, failed: rc = -5 [ 5067.443668] LustreError: 14533:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880130776680 x1834598193644928/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5067.451185] LustreError: 23676:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5067.452926] LustreError: 23676:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5067.504827] Lustre: server umount lustre-MDT0001 complete [ 5070.446582] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 5071.160316] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 5071.918786] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules [ 5076.026814] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5077.814817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5077.819201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5077.923678] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 5077.926100] Lustre: Failing over lustre-MDT0000 [ 5077.971106] LustreError: 25735:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5077.972807] LustreError: 25735:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5078.017266] Lustre: server umount lustre-MDT0000 complete [ 5078.018480] LustreError: 25735:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 5078.358720] Lustre: DEBUG MARKER: conf-sanity test_126: @@@@@@ FAIL: test_126 failed with 17 [ 5092.271589] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:54:45 (1749610485) [ 5096.619671] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 5102.064041] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5104.406125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5104.414888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5104.599100] LustreError: 28009:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 5104.604014] LustreError: 28009:0:(obd_config.c:1993:class_config_llog_handler()) Skipped 5 previous similar messages [ 5104.608362] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 5104.613326] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5104.621714] LustreError: Skipped 5 previous similar messages [ 5104.624382] LustreError: 27973:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 5104.629240] LustreError: 27973:0:(tgt_mount.c:1523:server_start_targets()) Skipped 5 previous similar messages [ 5104.633584] Lustre: Failing over lustre-MDT0000 [ 5104.708219] LustreError: 27973:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5104.712957] LustreError: 27973:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5104.772869] Lustre: server umount lustre-MDT0000 complete [ 5104.774381] LustreError: 27973:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 5105.216179] Lustre: DEBUG MARKER: conf-sanity test_127: @@@@@@ FAIL: MDT start failed [ 5107.020580] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:55:00 (1749610500) [ 5107.603915] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 5108.205920] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:55:01 (1749610501) [ 5115.010113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5115.196973] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 5115.199847] Lustre: Failing over lustre-MDT0000 [ 5115.283175] LustreError: 29731:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5115.287020] LustreError: 29731:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5115.362880] Lustre: server umount lustre-MDT0000 complete [ 5115.365125] LustreError: 29731:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 5115.934032] Lustre: DEBUG MARKER: conf-sanity test_129: @@@@@@ FAIL: MGS/MDS start failed [ 5117.825541] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:55:11 (1749610511) [ 5122.606266] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5125.779537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5125.952822] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 5125.958825] Lustre: Failing over lustre-MDT0000 [ 5126.101193] Lustre: server umount lustre-MDT0000 complete [ 5126.103858] LustreError: 31330:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 5128.594166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5145.727867] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5151.733761] LustreError: 31827:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.144@tcp: failed processing log, type 4: rc = -110 [ 5169.811894] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5169.815675] Lustre: 31827:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5177.794957] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5192.829646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5193.939920] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5195.009324] LustreError: 32364:0:(obd_class.h:1230:obd_iocontrol()) MGC192.168.203.144@tcp: no obd_iocontrol operation [ 5195.014639] LustreError: 32364:0:(obd_class.h:1230:obd_iocontrol()) Skipped 2 previous similar messages [ 5197.398957] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5197.405442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5243.516075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5245.371717] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5248.592908] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5248.598527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5279.623674] LustreError: lustre-OST0001: cannot register this server with the MGS: rc = -110. Is the MGS running? [ 5279.626763] LustreError: 1439:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -110 [ 5279.630208] LustreError: 1439:0:(tgt_mount.c:2268:server_fill_super()) Skipped 8 previous similar messages [ 5279.633416] LustreError: 1439:0:(tgt_mount.c:1762:server_put_super()) no obd lustre-OST0001 [ 5279.635933] LustreError: 1439:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0001 not registered [ 5279.641270] Lustre: server umount lustre-OST0001 complete [ 5279.643301] LustreError: 1439:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -110 [ 5397.789336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 5397.793999] LustreError: Skipped 8 previous similar messages [ 5512.971415] LustreError: 31846:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5512.982645] LustreError: 31846:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 291 previous similar messages [ 5520.289065] Lustre: 31840:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 5520.293703] Lustre: 31840:0:(mgc_request.c:1854:mgc_process_log()) Skipped 3 previous similar messages [ 5606.763792] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5608.037573] LustreError: 2184:0:(obd_class.h:1230:obd_iocontrol()) MGC192.168.203.144@tcp: no obd_iocontrol operation [ 5613.765082] LustreError: 31924:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000009:0x0:0x0]: rc = -5 [ 5613.771560] LustreError: 31924:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't get id from catalogs: rc = -5 [ 5613.777253] LustreError: 31924:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0000-osp-MDT0001: get update log duration 421, retries 0, failed: rc = -5 [ 5613.862573] LustreError: 2320:0:(obd_class.h:479:obd_check_dev()) Device 6 not setup [ 5613.865597] LustreError: 2320:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5613.923888] Lustre: server umount lustre-MDT0001 complete [ 5615.017482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5617.537381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5618.139711] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5635.568097] LustreError: 31840:0:(mgc_request.c:614:do_requeue()) failed processing log: -110 [ 5635.572216] LustreError: 31840:0:(mgc_request.c:614:do_requeue()) Skipped 31 previous similar messages [ 5648.642839] LustreError: lustre-MDT0001: cannot register this server with the MGS: rc = -110. Is the MGS running? [ 5648.648278] LustreError: 2985:0:(tgt_mount.c:2268:server_fill_super()) Unable to start targets: -110 [ 5648.652755] LustreError: 2985:0:(tgt_mount.c:1762:server_put_super()) no obd lustre-MDT0001 [ 5648.656922] LustreError: 2985:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0001 not registered [ 5648.736616] Lustre: server umount lustre-MDT0001 complete [ 5648.739370] LustreError: 2985:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -110 [ 5649.311779] Lustre: DEBUG MARKER: conf-sanity test_130: @@@@@@ FAIL: mdt2 start failed [ 5651.865267] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 23:04:05 (1749611045) [ 5657.294431] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5660.583178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5660.737592] LustreError: 4694:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 5660.740730] LustreError: 4694:0:(obd_config.c:1993:class_config_llog_handler()) Skipped 2 previous similar messages [ 5660.742944] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 5660.745741] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5660.754595] LustreError: Skipped 2 previous similar messages [ 5660.757499] LustreError: 4667:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -17 [ 5660.762425] LustreError: 4667:0:(tgt_mount.c:1523:server_start_targets()) Skipped 2 previous similar messages [ 5660.767207] Lustre: Failing over lustre-MDT0000 [ 5660.899817] Lustre: server umount lustre-MDT0000 complete [ 5660.901888] LustreError: 4667:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 5663.454645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5677.852669] Lustre: 31840:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1749611056/real 1749611056] req@ffff880096c1aa00 x1834598193708672/t0(0) o101->MGC192.168.203.144@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1749611072 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5677.868456] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5685.874829] LustreError: lustre-MDT0001: cannot register this server with the MGS: rc = -108. Is the MGS running? [ 5685.880373] LustreError: 5155:0:(tgt_mount.c:1762:server_put_super()) no obd lustre-MDT0001 [ 5685.884568] LustreError: 5155:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0001 not registered [ 5685.960685] Lustre: server umount lustre-MDT0001 complete [ 5685.963597] LustreError: 5155:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -108 [ 5692.091155] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5692.097951] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5700.125271] Lustre: server umount lustre-OST0001 complete [ 6027.185637] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6028.475782] LustreError: 7030:0:(obd_class.h:1230:obd_iocontrol()) MGC192.168.203.144@tcp: no obd_iocontrol operation [ 6028.482808] LustreError: 7030:0:(obd_class.h:1230:obd_iocontrol()) Skipped 1 previous similar message [ 6034.079598] Lustre: DEBUG MARKER: conf-sanity test_131: @@@@@@ FAIL: set dir project id failed