== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:09:39 (1773943779) start mds service on oleg317-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg317-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 532 (time 1773943838.36 total 10.14 last 52.47) - open/close 993 (time 1773943848.66 total 20.45 last 44.73) - open/close 1484 (time 1773943858.77 total 30.56 last 48.56) - open/close 2091 (time 1773943868.78 total 40.56 last 60.68) - open/close 2546 (time 1773943878.93 total 50.72 last 44.80) - open/close 2980 (time 1773943889.75 total 61.53 last 40.13) - open/close 3432 (time 1773943899.75 total 71.53 last 45.19) - open/close 3923 (time 1773943910.16 total 81.95 last 47.14) - open/close 4520 (time 1773943920.92 total 92.70 last 55.51) - open/close 5043 (time 1773943931.41 total 103.19 last 49.85) - open/close 5531 (time 1773943941.66 total 113.44 last 47.63) - open/close 6129 (time 1773943952.47 total 124.25 last 55.31) - open/close 6618 (time 1773943962.84 total 134.62 last 47.17) - open/close 7235 (time 1773943973.20 total 144.99 last 59.51) - open/close 7512 (time 1773943983.45 total 155.23 last 27.04) - open/close 7855 (time 1773943994.20 total 165.98 last 31.90) - open/close 8279 (time 1773944004.76 total 176.54 last 40.17) - open/close 8802 (time 1773944015.19 total 186.97 last 50.14) - open/close 9435 (time 1773944025.94 total 197.72 last 58.88) - open/close 10000 (time 1773944032.44 total 204.23 last 86.89) - open/close 10756 (time 1773944042.58 total 214.36 last 74.57) - open/close 11537 (time 1773944052.58 total 224.37 last 78.08) - open/close 12144 (time 1773944063.51 total 235.30 last 55.52) - open/close 12440 (time 1773944073.69 total 245.47 last 29.09) - open/close 12704 (time 1773944084.18 total 255.97 last 25.16) - open/close 13037 (time 1773944095.19 total 266.97 last 30.26) - open/close 13293 (time 1773944105.68 total 277.47 last 24.39) - open/close 13996 (time 1773944115.92 total 287.71 last 68.65) - open/close 14677 (time 1773944126.23 total 298.01 last 66.08) - open/close 15345 (time 1773944136.53 total 308.32 last 64.83) - open/close 16026 (time 1773944147.03 total 318.81 last 64.87) - open/close 16728 (time 1773944157.03 total 328.81 last 70.19) - open/close 17321 (time 1773944168.08 total 339.87 last 53.65) - open/close 17899 (time 1773944179.03 total 350.81 last 52.80) - open/close 18397 (time 1773944189.03 total 360.82 last 49.79) - open/close 20000 (time 1773944198.83 total 370.62 last 163.54) - open/close 22931 (time 1773944208.84 total 380.62 last 292.92) - open/close 25822 (time 1773944218.84 total 390.63 last 289.04) - open/close 29369 (time 1773944228.84 total 400.63 last 354.68) - open/close 32906 (time 1773944238.84 total 410.63 last 353.69) - open/close 36338 (time 1773944248.84 total 420.63 last 343.18) - open/close 40000 (time 1773944258.63 total 430.42 last 374.08) - open/close 43425 (time 1773944268.63 total 440.42 last 342.46) - open/close 46528 (time 1773944278.63 total 450.42 last 310.26) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 458.72 seconds: 108.20 ops/second - unlinked 0 (time 1773944288 ; total 0 ; last 0) - unlinked 10000 (time 1773944300 ; total 12 ; last 12) - unlinked 20000 (time 1773944312 ; total 24 ; last 12) - unlinked 30000 (time 1773944325 ; total 37 ; last 13) - unlinked 40000 (time 1773944340 ; total 52 ; last 15) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 64 seconds: 775.500000 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local oleg317-server: modules unloaded. pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2