== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:03:44 (1775045024) start mds service on oleg150-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing load_modules_local oleg150-server: Loading modules from /home/green/git/lustre-release/lustre oleg150-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg150-server: detected 4 online CPUs by sysfs oleg150-server: MODOPTS_LIBCFS= oleg150-server: Force libcfs to create 2 CPU partitions oleg150-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg150-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg150-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg150-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg150-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 541 (time 1775045065.39 total 10.38 last 52.13) - open/close 947 (time 1775045075.50 total 20.49 last 40.15) - open/close 1406 (time 1775045085.61 total 30.61 last 45.37) - open/close 2078 (time 1775045096.23 total 41.23 last 63.28) - open/close 2496 (time 1775045106.60 total 51.60 last 40.31) - open/close 3286 (time 1775045116.60 total 61.60 last 78.98) - open/close 3831 (time 1775045127.08 total 72.08 last 52.01) - open/close 4386 (time 1775045137.51 total 82.51 last 53.21) - open/close 4934 (time 1775045148.08 total 93.07 last 51.87) - open/close 5415 (time 1775045158.38 total 103.37 last 46.70) - open/close 6016 (time 1775045169.00 total 113.99 last 56.57) - open/close 6772 (time 1775045179.95 total 124.94 last 69.05) - open/close 7639 (time 1775045190.32 total 135.31 last 83.63) - open/close 8419 (time 1775045200.32 total 145.31 last 77.98) - open/close 9215 (time 1775045211.38 total 156.37 last 72.00) - open/close 9926 (time 1775045222.12 total 167.11 last 66.16) - open/close 10698 (time 1775045232.23 total 177.23 last 76.35) - open/close 11433 (time 1775045242.23 total 187.23 last 73.50) - open/close 12291 (time 1775045252.59 total 197.58 last 82.83) - open/close 13192 (time 1775045262.89 total 207.88 last 87.48) - open/close 13954 (time 1775045272.94 total 217.93 last 75.82) - open/close 14762 (time 1775045283.24 total 228.24 last 78.42) - open/close 15689 (time 1775045293.68 total 238.67 last 88.84) - open/close 16492 (time 1775045303.92 total 248.92 last 78.38) - open/close 17207 (time 1775045314.92 total 259.92 last 65.00) - open/close 17715 (time 1775045325.23 total 270.22 last 49.31) - open/close 18454 (time 1775045335.23 total 280.22 last 73.90) - open/close 19246 (time 1775045346.22 total 291.22 last 72.02) - open/close 20000 (time 1775045355.26 total 300.25 last 83.47) - open/close 23270 (time 1775045365.26 total 310.25 last 326.95) - open/close 27750 (time 1775045375.26 total 320.25 last 447.98) - open/close 30000 (time 1775045380.20 total 325.19 last 455.73) - open/close 34556 (time 1775045390.20 total 335.19 last 455.59) - open/close 39224 (time 1775045400.20 total 345.19 last 466.78) - open/close 43781 (time 1775045410.20 total 355.19 last 455.65) - open/close 48165 (time 1775045420.20 total 365.19 last 438.31) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 368.54 seconds: 134.67 ops/second - unlinked 0 (time 1775045424 ; total 0 ; last 0) - unlinked 10000 (time 1775045437 ; total 13 ; last 13) - unlinked 20000 (time 1775045448 ; total 24 ; last 11) - unlinked 30000 (time 1775045459 ; total 35 ; last 11) - unlinked 40000 (time 1775045471 ; total 47 ; last 12) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 60 seconds: 827.200012 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg150-server unloading modules via unload_modules_local on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing unload_modules_local oleg150-server: modules unloaded. pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2