== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:29:28 (1786570168) start mds service on oleg135-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions oleg135-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg135-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg135-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg135-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg135-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 267 (time 1786570232.84 total 10.65 last 25.08) - open/close 566 (time 1786570242.95 total 20.76 last 29.57) - open/close 916 (time 1786570252.96 total 30.77 last 34.97) - open/close 1269 (time 1786570263.50 total 41.31 last 33.49) - open/close 1575 (time 1786570274.25 total 52.06 last 28.46) - open/close 1729 (time 1786570284.55 total 62.36 last 14.95) - open/close 2010 (time 1786570294.80 total 72.60 last 27.43) - open/close 2232 (time 1786570304.84 total 82.65 last 22.11) - open/close 2471 (time 1786570315.02 total 92.83 last 23.48) - open/close 2744 (time 1786570325.33 total 103.14 last 26.48) - open/close 2950 (time 1786570335.95 total 113.76 last 19.39) - open/close 3221 (time 1786570345.96 total 123.76 last 27.10) - open/close 3453 (time 1786570355.96 total 133.77 last 23.19) - open/close 3763 (time 1786570366.74 total 144.54 last 28.76) - open/close 4082 (time 1786570376.75 total 154.56 last 31.86) - open/close 4348 (time 1786570388.17 total 165.98 last 23.29) - open/close 4613 (time 1786570398.35 total 176.16 last 26.04) - open/close 4888 (time 1786570408.84 total 186.65 last 26.21) - open/close 5092 (time 1786570419.27 total 197.08 last 19.56) - open/close 5266 (time 1786570429.51 total 207.32 last 16.99) - open/close 5639 (time 1786570440.33 total 218.14 last 34.48) - open/close 5907 (time 1786570450.34 total 228.15 last 26.78) - open/close 6222 (time 1786570461.27 total 239.07 last 28.84) - open/close 6452 (time 1786570472.20 total 250.01 last 21.03) - open/close 6723 (time 1786570482.25 total 260.05 last 26.97) - open/close 6982 (time 1786570492.50 total 270.30 last 25.27) - open/close 7256 (time 1786570504.01 total 281.82 last 23.80) - open/close 7502 (time 1786570514.44 total 292.25 last 23.58) - open/close 7648 (time 1786570524.68 total 302.49 last 14.26) - open/close 7895 (time 1786570535.24 total 313.05 last 23.39) - open/close 8105 (time 1786570545.93 total 323.74 last 19.64) - open/close 8448 (time 1786570556.87 total 334.68 last 31.36) - open/close 8671 (time 1786570567.64 total 345.44 last 20.71) - open/close 9009 (time 1786570579.61 total 357.42 last 28.23) - open/close 9304 (time 1786570590.09 total 367.90 last 28.14) - open/close 9544 (time 1786570600.14 total 377.94 last 23.90) - open/close 9802 (time 1786570610.14 total 387.94 last 25.80) - open/close 10000 (time 1786570617.61 total 395.41 last 26.51) - open/close 10184 (time 1786570629.00 total 406.81 last 16.15) - open/close 10351 (time 1786570639.11 total 416.92 last 16.51) - open/close 13177 (time 1786570649.12 total 426.92 last 282.54) - open/close 15914 (time 1786570659.12 total 436.93 last 273.60) - open/close 18113 (time 1786570669.12 total 446.93 last 219.84) - open/close 20000 (time 1786570677.17 total 454.98 last 234.45) - open/close 22569 (time 1786570687.17 total 464.98 last 256.82) - open/close 25871 (time 1786570697.18 total 474.98 last 330.12) - open/close 29885 (time 1786570707.18 total 484.99 last 401.34) - open/close 33759 (time 1786570717.18 total 494.99 last 387.38) - open/close 36447 (time 1786570727.18 total 504.99 last 268.80) - open/close 39124 (time 1786570737.18 total 514.99 last 267.61) - open/close 40000 (time 1786570740.46 total 518.27 last 267.11) - open/close 43569 (time 1786570750.46 total 528.27 last 356.86) - open/close 46311 (time 1786570760.46 total 538.27 last 274.16) - open/close 49039 (time 1786570770.47 total 548.27 last 272.78) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 550.83 seconds: 90.10 ops/second - unlinked 0 (time 1786570774 ; total 0 ; last 0) - unlinked 10000 (time 1786570791 ; total 17 ; last 17) - unlinked 20000 (time 1786570807 ; total 33 ; last 16) - unlinked 30000 (time 1786570826 ; total 52 ; last 19) - unlinked 40000 (time 1786570840 ; total 66 ; last 14) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 81 seconds: 612.728394 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg135-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server stop mds service on oleg135-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server stop mds service on oleg135-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg135-server unloading modules via unload_modules_local on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing unload_modules_local oleg135-server: modules unloaded. pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2