== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:35:49 (1775169349) start mds service on oleg348-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing load_modules_local oleg348-server: Loading modules from /home/green/git/lustre-release/lustre oleg348-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg348-server: detected 4 online CPUs by sysfs oleg348-server: MODOPTS_LIBCFS= oleg348-server: Force libcfs to create 2 CPU partitions oleg348-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg348-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg348-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg348-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg348-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-OST0000 oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg348-client.virtnet: -o user_xattr,flock 192.168.203.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 719 (time 1775169407.84 total 10.14 last 70.93) - open/close 1420 (time 1775169418.02 total 20.31 last 68.89) - open/close 2012 (time 1775169429.03 total 31.32 last 53.77) - open/close 2551 (time 1775169439.03 total 41.33 last 53.89) - open/close 3201 (time 1775169449.32 total 51.61 last 63.20) - open/close 3873 (time 1775169459.43 total 61.72 last 66.47) - open/close 4579 (time 1775169469.43 total 71.72 last 70.60) - open/close 5306 (time 1775169479.43 total 81.72 last 72.69) - open/close 5940 (time 1775169489.43 total 91.72 last 63.39) - open/close 6648 (time 1775169499.43 total 101.73 last 70.79) - open/close 7352 (time 1775169509.47 total 111.77 last 70.09) - open/close 8145 (time 1775169520.48 total 122.78 last 72.03) - open/close 8926 (time 1775169530.79 total 133.09 last 75.78) - open/close 9634 (time 1775169540.96 total 143.26 last 69.59) - open/close 10000 (time 1775169546.11 total 148.41 last 71.10) - open/close 10734 (time 1775169556.11 total 158.41 last 73.40) - open/close 11413 (time 1775169567.01 total 169.31 last 62.29) - open/close 12066 (time 1775169577.89 total 180.18 last 60.03) - open/close 12710 (time 1775169587.89 total 190.19 last 64.39) - open/close 13360 (time 1775169598.63 total 200.92 last 60.53) - open/close 14023 (time 1775169608.63 total 210.93 last 66.28) - open/close 14667 (time 1775169619.04 total 221.34 last 61.86) - open/close 15202 (time 1775169629.09 total 231.39 last 53.24) - open/close 15737 (time 1775169639.91 total 242.20 last 49.47) - open/close 16471 (time 1775169649.96 total 252.25 last 73.02) - open/close 17180 (time 1775169659.96 total 262.26 last 70.87) - open/close 17795 (time 1775169669.96 total 272.26 last 61.49) - open/close 18395 (time 1775169679.96 total 282.26 last 60.00) - open/close 19028 (time 1775169690.73 total 293.02 last 58.81) - open/close 19530 (time 1775169702.57 total 304.86 last 42.40) - open/close 20000 (time 1775169708.98 total 311.27 last 73.30) - open/close 22623 (time 1775169718.98 total 321.28 last 262.22) - open/close 27593 (time 1775169728.98 total 331.28 last 496.93) - open/close 30000 (time 1775169733.49 total 335.78 last 534.29) - open/close 35145 (time 1775169743.49 total 345.78 last 514.44) - open/close 40000 (time 1775169752.80 total 355.10 last 521.20) - open/close 45245 (time 1775169762.81 total 365.10 last 524.47) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 373.66 seconds: 132.83 ops/second - unlinked 0 (time 1775169772 ; total 0 ; last 0) - unlinked 10000 (time 1775169784 ; total 12 ; last 12) - unlinked 20000 (time 1775169794 ; total 22 ; last 10) - unlinked 30000 (time 1775169806 ; total 34 ; last 12) - unlinked 40000 (time 1775169817 ; total 45 ; last 11) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 57 seconds: 870.736816 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg348-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg348-server unloading modules via unload_modules_local on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing unload_modules_local oleg348-server: modules unloaded. pdsh@oleg348-client: oleg348-client: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-client: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2