== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:51:24 (1778280684) start mds service on oleg116-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg116-server: detected 4 online CPUs by sysfs oleg116-server: MODOPTS_LIBCFS= oleg116-server: Force libcfs to create 2 CPU partitions oleg116-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg116-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg116-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg116-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg116-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 648 (time 1778280726.51 total 10.76 last 60.22) - open/close 1368 (time 1778280737.52 total 21.77 last 65.41) - open/close 1955 (time 1778280748.21 total 32.46 last 54.92) - open/close 2609 (time 1778280758.96 total 43.21 last 60.83) - open/close 3162 (time 1778280769.45 total 53.70 last 52.69) - open/close 3918 (time 1778280779.45 total 63.70 last 75.60) - open/close 4574 (time 1778280790.32 total 74.57 last 60.38) - open/close 5320 (time 1778280800.32 total 84.57 last 74.59) - open/close 6067 (time 1778280810.32 total 94.57 last 74.68) - open/close 6845 (time 1778280820.46 total 104.71 last 76.73) - open/close 7600 (time 1778280830.46 total 114.71 last 75.49) - open/close 8207 (time 1778280841.13 total 125.38 last 56.88) - open/close 8924 (time 1778280852.01 total 136.27 last 65.89) - open/close 9706 (time 1778280862.02 total 146.27 last 78.19) - open/close 10000 (time 1778280866.81 total 151.07 last 61.26) - open/close 10727 (time 1778280877.74 total 161.99 last 66.54) - open/close 11403 (time 1778280889.52 total 173.77 last 57.39) - open/close 12181 (time 1778280899.56 total 183.82 last 77.46) - open/close 12805 (time 1778280910.25 total 194.51 last 58.37) - open/close 13560 (time 1778280920.26 total 204.51 last 75.49) - open/close 14291 (time 1778280931.25 total 215.50 last 66.52) - open/close 14913 (time 1778280941.94 total 226.19 last 58.18) - open/close 15669 (time 1778280952.88 total 237.13 last 69.10) - open/close 16151 (time 1778280963.31 total 247.56 last 46.20) - open/close 16781 (time 1778280974.07 total 258.32 last 58.57) - open/close 17568 (time 1778280984.24 total 268.49 last 77.35) - open/close 18205 (time 1778280994.99 total 279.24 last 59.24) - open/close 18564 (time 1778281005.75 total 290.00 last 33.38) - open/close 18973 (time 1778281016.30 total 300.56 last 38.73) - open/close 19699 (time 1778281026.31 total 310.56 last 72.59) - open/close 20000 (time 1778281030.07 total 314.33 last 79.89) - open/close 23766 (time 1778281040.08 total 324.33 last 376.44) - open/close 27408 (time 1778281050.08 total 334.33 last 364.18) - open/close 30000 (time 1778281057.16 total 341.41 last 366.13) - open/close 32685 (time 1778281067.16 total 351.41 last 268.46) - open/close 35869 (time 1778281077.16 total 361.41 last 318.39) - open/close 39473 (time 1778281087.16 total 371.41 last 360.34) - open/close 43241 (time 1778281097.16 total 381.42 last 376.72) - open/close 46331 (time 1778281107.17 total 391.42 last 308.95) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 399.81 seconds: 124.14 ops/second - unlinked 0 (time 1778281117 ; total 0 ; last 0) - unlinked 10000 (time 1778281131 ; total 14 ; last 14) - unlinked 20000 (time 1778281145 ; total 28 ; last 14) - unlinked 30000 (time 1778281158 ; total 41 ; last 13) - unlinked 40000 (time 1778281169 ; total 52 ; last 11) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 65 seconds: 763.569214 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg116-server unloading modules via unload_modules_local on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing unload_modules_local oleg116-server: modules unloaded. pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2