== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:19:03 (1787858343) start mds service on oleg423-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing load_modules_local oleg423-server: Loading modules from /home/green/git/lustre-release/lustre oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg423-server: detected 4 online CPUs by sysfs oleg423-server: MODOPTS_LIBCFS= oleg423-server: Force libcfs to create 2 CPU partitions oleg423-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg423-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg423-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg423-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg423-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 135 (time 1787858471.81 total 10.52 last 12.83) - open/close 268 (time 1787858482.71 total 21.42 last 12.20) - open/close 349 (time 1787858492.74 total 31.45 last 8.08) - open/close 535 (time 1787858502.75 total 41.45 last 18.59) - open/close 727 (time 1787858514.02 total 52.73 last 17.03) - open/close 877 (time 1787858524.37 total 63.08 last 14.49) - open/close 1106 (time 1787858534.38 total 73.09 last 22.88) - open/close 1227 (time 1787858544.40 total 83.11 last 12.07) - open/close 1401 (time 1787858555.06 total 93.77 last 16.32) - open/close 1556 (time 1787858565.45 total 104.16 last 14.92) - open/close 1823 (time 1787858575.45 total 114.16 last 26.69) - open/close 2084 (time 1787858585.46 total 124.17 last 26.08) - open/close 2278 (time 1787858595.47 total 134.18 last 19.39) - open/close 2508 (time 1787858605.85 total 144.56 last 22.15) - open/close 2758 (time 1787858615.85 total 154.56 last 24.99) - open/close 2997 (time 1787858626.56 total 165.27 last 22.32) - open/close 3248 (time 1787858636.57 total 175.27 last 25.08) - open/close 3383 (time 1787858646.59 total 185.30 last 13.47) - open/close 3643 (time 1787858657.09 total 195.80 last 24.75) - open/close 3886 (time 1787858669.91 total 208.62 last 18.96) - open/close 4120 (time 1787858680.06 total 218.77 last 23.06) - open/close 4385 (time 1787858690.06 total 228.77 last 26.49) - open/close 4624 (time 1787858700.58 total 239.29 last 22.72) - open/close 4695 (time 1787858710.77 total 249.48 last 6.96) - open/close 4864 (time 1787858720.78 total 259.49 last 16.89) - open/close 4976 (time 1787858730.79 total 269.50 last 11.19) - open/close 5126 (time 1787858742.88 total 281.59 last 12.40) - open/close 5260 (time 1787858754.14 total 292.85 last 11.90) - open/close 5485 (time 1787858764.84 total 303.55 last 21.04) - open/close 5651 (time 1787858775.91 total 314.62 last 14.99) - open/close 5893 (time 1787858785.94 total 324.65 last 24.14) - open/close 6121 (time 1787858796.41 total 335.12 last 21.77) - open/close 6346 (time 1787858806.42 total 345.13 last 22.49) - open/close 6610 (time 1787858816.43 total 355.14 last 26.35) - open/close 6861 (time 1787858827.46 total 366.17 last 22.76) - open/close 7054 (time 1787858838.23 total 376.94 last 17.92) - open/close 7263 (time 1787858848.23 total 386.94 last 20.90) - open/close 7525 (time 1787858858.24 total 396.95 last 26.17) - open/close 7756 (time 1787858868.67 total 407.37 last 22.17) - open/close 8011 (time 1787858879.43 total 418.14 last 23.70) - open/close 8237 (time 1787858889.62 total 428.33 last 22.17) - open/close 8444 (time 1787858900.25 total 438.96 last 19.48) - open/close 8652 (time 1787858911.16 total 449.87 last 19.06) - open/close 8879 (time 1787858921.92 total 460.63 last 21.10) - open/close 9129 (time 1787858932.62 total 471.33 last 23.37) - open/close 9402 (time 1787858942.62 total 481.33 last 27.29) - open/close 9632 (time 1787858953.16 total 491.87 last 21.81) - open/close 9822 (time 1787858963.27 total 501.98 last 18.81) - open/close 9929 (time 1787858975.76 total 514.47 last 8.57) - open/close 10000 (time 1787858979.48 total 518.19 last 19.09) - open/close 10089 (time 1787858991.82 total 530.53 last 7.21) - open/close 10280 (time 1787859002.13 total 540.84 last 18.54) - open/close 10488 (time 1787859012.14 total 550.85 last 20.77) - open/close 10609 (time 1787859022.15 total 560.86 last 12.09) - open/close 10720 (time 1787859033.03 total 571.74 last 10.20) - open/close 10841 (time 1787859045.18 total 583.89 last 9.96) - open/close 11049 (time 1787859056.09 total 594.80 last 19.08) - open/close 11291 (time 1787859066.09 total 604.80 last 24.19) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-11409) error: No space left on device total: 11409 open/close in 609.07 seconds: 18.73 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (114882) - No such process - unlinked 0 (time 1787859076 ; total 0 ; last 0) - unlinked 10000 (time 1787859161 ; total 85 ; last 85) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-11409) error: No such file or directory total: 11409 unlinks in 96 seconds: 118.843750 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg423-server unloading modules via unload_modules_local on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing unload_modules_local oleg423-server: modules unloaded. pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2