== sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:27:26 (1788971246) Setting default values and create backup for check Check default configuration found: net.ipv4.neigh.default.gc_thresh1 128 found: net.ipv4.neigh.default.gc_thresh2 512 found: net.ipv4.neigh.default.gc_thresh3 1024 found: net.ipv6.neigh.default.gc_thresh1 128 found: net.ipv6.neigh.default.gc_thresh2 512 found: net.ipv6.neigh.default.gc_thresh3 1024 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions enable_sysctl_setup set to 0 New configuration enable_sysctl_setup set to 1 /home/green/git/lustre-release/lustre/scripts/lnet-sysctl-config: line 28: /etc/lnet-sysctl.conf: No such file or directory Check new configuration /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh: line 5967: /etc/lnet-sysctl.conf: No such file or directory Reset to original values enable_sysctl_setup set to 0 Check original configuration found: net.ipv4.neigh.default.gc_thresh1 128 found: net.ipv4.neigh.default.gc_thresh2 512 found: net.ipv4.neigh.default.gc_thresh3 1024 found: net.ipv6.neigh.default.gc_thresh1 128 found: net.ipv6.neigh.default.gc_thresh2 512 found: net.ipv6.neigh.default.gc_thresh3 1024 lustre_rmmod: wait 41s for 7 OBD devices lustre_rmmod: OBD devices after 60s: 0 UP mgc MGC192.168.203.150@tcp eebf8878-e019-48cb-9caf-f6cb96348888 5 1 UP lov lustre-clilov-ffff9a73d1aa9000 e6d587aa-ca1e-4fb4-bf99-a5b204dc5e6f 4 2 UP lmv lustre-clilmv-ffff9a73d1aa9000 e6d587aa-ca1e-4fb4-bf99-a5b204dc5e6f 5 3 UP mdc lustre-MDT0000-mdc-ffff9a73d1aa9000 e6d587aa-ca1e-4fb4-bf99-a5b204dc5e6f 5 4 UP mdc lustre-MDT0001-mdc-ffff9a73d1aa9000 e6d587aa-ca1e-4fb4-bf99-a5b204dc5e6f 5 5 UP osc lustre-OST0000-osc-ffff9a73d1aa9000 e6d587aa-ca1e-4fb4-bf99-a5b204dc5e6f 5 6 UP osc lustre-OST0001-osc-ffff9a73d1aa9000 e6d587aa-ca1e-4fb4-bf99-a5b204dc5e6f 5 rmmod: ERROR: Module mdd is in use unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: lustre_rmmod: wait 41s for 29 OBD devices oleg350-server: lustre_rmmod: OBD devices after 60s: oleg350-server: 0 UP osd-ldiskfs lustre-MDT0000-osd lustre-MDT0000-osd_UUID 11 oleg350-server: 1 UP mgs MGS MGS 7 oleg350-server: 2 UP mgc MGC192.168.203.150@tcp c9797fc9-9356-4874-bccd-5c23fcc6bf14 5 oleg350-server: 3 UP mds MDS MDS_uuid 3 oleg350-server: 4 UP lod lustre-MDT0000-mdtlov lustre-MDT0000-mdtlov_UUID 4 oleg350-server: 5 UP mdt lustre-MDT0000 lustre-MDT0000_UUID 15 oleg350-server: 6 UP mdd lustre-MDD0000 lustre-MDD0000_UUID 4 oleg350-server: 7 UP qmt lustre-QMT0000 lustre-QMT0000_UUID 4 oleg350-server: 8 UP osp lustre-MDT0001-osp-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg350-server: 9 UP osp lustre-OST0000-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg350-server: 10 UP osp lustre-OST0001-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg350-server: 11 UP lwp lustre-MDT0000-lwp-MDT0000 lustre-MDT0000-lwp-MDT0000_UUID 5 oleg350-server: 12 UP osd-ldiskfs lustre-MDT0001-osd lustre-MDT0001-osd_UUID 9 oleg350-server: 13 UP lod lustre-MDT0001-mdtlov lustre-MDT0001-mdtlov_UUID 4 oleg350-server: 14 UP mdt lustre-MDT0001 lustre-MDT0001_UUID 11 oleg350-server: 15 UP mdd lustre-MDD0001 lustre-MDD0001_UUID 4 oleg350-server: 16 UP osp lustre-MDT0000-osp-MDT0001 lustre-MDT0001-mdtlov_UUID 5 oleg350-server: 17 UP osp lustre-OST0000-osc-MDT0001 lustre-MDT0001-mdtlov_UUID 5 oleg350-server: 18 UP osp lustre-OST0001-osc-MDT0001 lustre-MDT0001-mdtlov_UUID 5 oleg350-server: 19 UP lwp lustre-MDT0000-lwp-MDT0001 lustre-MDT0000-lwp-MDT0001_UUID 5 oleg350-server: 20 UP osd-ldiskfs lustre-OST0000-osd lustre-OST0000-osd_UUID 5 oleg350-server: 21 UP ost OSS OSS_uuid 3 oleg350-server: 22 UP obdfilter lustre-OST0000 lustre-OST0000_UUID 7 oleg350-server: 23 UP lwp lustre-MDT0000-lwp-OST0000 lustre-MDT0000-lwp-OST0000_UUID 5 oleg350-server: 24 UP lwp lustre-MDT0001-lwp-OST0000 lustre-MDT0001-lwp-OST0000_UUID 5 oleg350-server: 25 UP osd-ldiskfs lustre-OST0001-osd lustre-OST0001-osd_UUID 5 oleg350-server: 26 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 7 oleg350-server: 27 UP lwp lustre-MDT0000-lwp-OST0001 lustre-MDT0000-lwp-OST0001_UUID 5 oleg350-server: 28 UP lwp lustre-MDT0001-lwp-OST0001 lustre-MDT0001-lwp-OST0001_UUID 5 oleg350-server: oleg350-server: rmmod: ERROR: Module mdd is in use pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 sanity-lnet test_260: @@@@@@ FAIL: Failed to cleanup Modules Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6067:test_260() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6069:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_260.*.1788971393.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_260.debug_log.oleg350-server.1788971393.log.yzvAwj" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_260.dmesg.oleg350-server.1788971393.log.pX67IW" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]