== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:43:38 (1783565018) start mds service on oleg407-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions oleg407-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg407-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg407-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg407-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg407-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 228 (time 1783565091.60 total 14.12 last 16.15) - open/close 612 (time 1783565102.42 total 24.94 last 35.49) - open/close 985 (time 1783565112.53 total 35.05 last 36.89) - open/close 1369 (time 1783565122.83 total 45.35 last 37.29) - open/close 1689 (time 1783565134.68 total 57.20 last 27.01) - open/close 2052 (time 1783565144.68 total 67.20 last 36.28) - open/close 2310 (time 1783565154.77 total 77.28 last 25.58) - open/close 2587 (time 1783565165.39 total 87.91 last 26.06) - open/close 2931 (time 1783565175.39 total 97.91 last 34.40) - open/close 3216 (time 1783565185.68 total 108.20 last 27.71) - open/close 3504 (time 1783565196.31 total 118.82 last 27.10) - open/close 3802 (time 1783565206.31 total 128.83 last 29.79) - open/close 3999 (time 1783565216.31 total 138.83 last 19.70) - open/close 4291 (time 1783565226.84 total 149.36 last 27.74) - open/close 4612 (time 1783565237.65 total 160.17 last 29.69) - open/close 4922 (time 1783565248.54 total 171.06 last 28.47) - open/close 5199 (time 1783565258.70 total 181.22 last 27.25) - open/close 5478 (time 1783565269.52 total 192.04 last 25.79) - open/close 5747 (time 1783565280.46 total 202.98 last 24.58) - open/close 6061 (time 1783565290.47 total 212.98 last 31.40) - open/close 6327 (time 1783565300.47 total 222.99 last 26.59) - open/close 6515 (time 1783565310.92 total 233.44 last 17.98) - open/close 6669 (time 1783565321.24 total 243.76 last 14.93) - open/close 6888 (time 1783565332.62 total 255.14 last 19.24) - open/close 6993 (time 1783565343.50 total 266.02 last 9.65) - open/close 7078 (time 1783565354.13 total 276.64 last 8.00) - open/close 7358 (time 1783565364.69 total 287.20 last 26.52) - open/close 7457 (time 1783565375.57 total 298.08 last 9.10) - open/close 7594 (time 1783565385.61 total 308.13 last 13.64) - open/close 7719 (time 1783565395.61 total 318.13 last 12.50) - open/close 7976 (time 1783565406.41 total 328.93 last 23.80) - open/close 8127 (time 1783565416.53 total 339.04 last 14.93) - open/close 8556 (time 1783565426.53 total 349.04 last 42.90) - open/close 9359 (time 1783565437.07 total 359.59 last 76.12) - open/close 9528 (time 1783565447.31 total 369.83 last 16.50) - open/close 9680 (time 1783565457.75 total 380.27 last 14.56) - open/close 9860 (time 1783565468.17 total 390.69 last 17.27) - open/close 10000 (time 1783565474.16 total 396.68 last 23.38) - open/close 10185 (time 1783565484.43 total 406.95 last 18.01) - open/close 10596 (time 1783565494.44 total 416.95 last 41.08) - open/close 12700 (time 1783565504.44 total 426.96 last 210.34) - open/close 15351 (time 1783565514.44 total 436.96 last 264.99) - open/close 18595 (time 1783565524.45 total 446.96 last 324.34) - open/close 20000 (time 1783565530.79 total 453.31 last 221.38) - open/close 22343 (time 1783565540.80 total 463.31 last 234.23) - open/close 24886 (time 1783565550.80 total 473.32 last 254.22) - open/close 27605 (time 1783565560.80 total 483.32 last 271.80) - open/close 30000 (time 1783565568.62 total 491.14 last 306.40) - open/close 32693 (time 1783565578.64 total 501.16 last 268.70) - open/close 35717 (time 1783565588.64 total 511.16 last 302.38) - open/close 38558 (time 1783565598.64 total 521.16 last 284.04) - open/close 40000 (time 1783565603.83 total 526.34 last 278.26) - open/close 43124 (time 1783565613.83 total 536.35 last 312.33) - open/close 46190 (time 1783565623.83 total 546.35 last 306.59) - open/close 48822 (time 1783565633.83 total 556.35 last 263.16) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 559.56 seconds: 88.70 ops/second - unlinked 0 (time 1783565638 ; total 0 ; last 0) - unlinked 10000 (time 1783565663 ; total 25 ; last 25) - unlinked 20000 (time 1783565688 ; total 50 ; last 25) - unlinked 30000 (time 1783565713 ; total 75 ; last 25) - unlinked 40000 (time 1783565742 ; total 104 ; last 29) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 130 seconds: 381.776917 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg407-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg407-server unloading modules via unload_modules_local on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing unload_modules_local oleg407-server: modules unloaded. pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2