== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:36:55 (1777498615) start mds service on oleg358-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing load_modules_local oleg358-server: Loading modules from /home/green/git/lustre-release/lustre oleg358-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg358-server: detected 4 online CPUs by sysfs oleg358-server: MODOPTS_LIBCFS= oleg358-server: Force libcfs to create 2 CPU partitions oleg358-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg358-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg358-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg358-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg358-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-OST0000 oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg358-client.virtnet: -o user_xattr,flock 192.168.203.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 146 (time 1777498687.06 total 10.01 last 14.59) - open/close 278 (time 1777498697.99 total 20.94 last 12.07) - open/close 358 (time 1777498708.30 total 31.25 last 7.76) - open/close 488 (time 1777498719.18 total 42.13 last 11.95) - open/close 706 (time 1777498729.18 total 52.13 last 21.79) - open/close 846 (time 1777498739.18 total 62.13 last 14.00) - open/close 990 (time 1777498749.19 total 72.14 last 14.39) - open/close 1095 (time 1777498759.81 total 82.76 last 9.89) - open/close 1204 (time 1777498770.31 total 93.26 last 10.39) - open/close 1305 (time 1777498780.93 total 103.88 last 9.50) - open/close 1464 (time 1777498790.94 total 113.89 last 15.89) - open/close 1814 (time 1777498800.94 total 123.89 last 35.00) - open/close 2041 (time 1777498811.35 total 134.30 last 21.81) - open/close 2215 (time 1777498821.35 total 144.30 last 17.39) - open/close 2377 (time 1777498831.36 total 154.31 last 16.19) - open/close 2583 (time 1777498841.49 total 164.44 last 20.33) - open/close 2770 (time 1777498852.53 total 175.48 last 16.94) - open/close 2955 (time 1777498863.45 total 186.40 last 16.94) - open/close 3144 (time 1777498874.54 total 197.49 last 17.04) - open/close 3342 (time 1777498884.64 total 207.59 last 19.62) - open/close 3565 (time 1777498894.66 total 217.61 last 22.25) - open/close 3680 (time 1777498907.54 total 230.49 last 8.93) - open/close 3867 (time 1777498917.91 total 240.86 last 18.02) - open/close 4138 (time 1777498928.78 total 251.73 last 24.93) - open/close 4372 (time 1777498938.80 total 261.75 last 23.36) - open/close 4646 (time 1777498948.82 total 271.77 last 27.36) - open/close 4890 (time 1777498958.98 total 281.93 last 24.00) - open/close 5137 (time 1777498968.99 total 291.94 last 24.69) - open/close 5368 (time 1777498979.67 total 302.62 last 21.63) - open/close 5571 (time 1777498989.67 total 312.62 last 20.29) - open/close 5742 (time 1777498999.88 total 322.83 last 16.75) - open/close 5936 (time 1777499010.69 total 333.64 last 17.94) - open/close 6128 (time 1777499020.70 total 343.65 last 19.18) - open/close 6332 (time 1777499031.11 total 354.06 last 19.60) - open/close 6538 (time 1777499041.12 total 364.07 last 20.59) - open/close 6772 (time 1777499051.98 total 374.93 last 21.55) - open/close 7014 (time 1777499062.53 total 385.48 last 22.92) - open/close 7155 (time 1777499072.54 total 395.49 last 14.09) - open/close 7376 (time 1777499083.20 total 406.15 last 20.72) - open/close 7621 (time 1777499094.31 total 417.26 last 22.06) - open/close 7837 (time 1777499104.77 total 427.72 last 20.65) - open/close 7998 (time 1777499114.82 total 437.77 last 16.02) - open/close 8237 (time 1777499124.83 total 447.78 last 23.88) - open/close 8464 (time 1777499134.84 total 457.79 last 22.68) - open/close 8661 (time 1777499144.85 total 467.80 last 19.67) - open/close 8860 (time 1777499155.08 total 478.03 last 19.46) - open/close 9037 (time 1777499165.38 total 488.33 last 17.18) - open/close 9172 (time 1777499176.32 total 499.27 last 12.34) - open/close 9312 (time 1777499187.27 total 510.22 last 12.78) - open/close 9467 (time 1777499197.38 total 520.33 last 15.34) - open/close 9652 (time 1777499207.89 total 530.84 last 17.59) - open/close 9741 (time 1777499218.37 total 541.32 last 8.49) - open/close 9949 (time 1777499229.13 total 552.08 last 19.34) - open/close 10000 (time 1777499231.83 total 554.78 last 18.88) - open/close 10187 (time 1777499242.63 total 565.58 last 17.31) - open/close 10323 (time 1777499252.63 total 575.58 last 13.60) - open/close 10622 (time 1777499262.63 total 585.58 last 29.88) - open/close 10834 (time 1777499273.55 total 596.50 last 19.43) - open/close 10975 (time 1777499283.72 total 606.67 last 13.86) - open/close 11125 (time 1777499293.73 total 616.68 last 14.99) - open/close 11314 (time 1777499304.53 total 627.48 last 17.50) - open/close 11512 (time 1777499314.56 total 637.51 last 19.73) - open/close 11625 (time 1777499325.32 total 648.27 last 10.50) - open/close 11727 (time 1777499335.76 total 658.71 last 9.78) - open/close 11853 (time 1777499345.76 total 668.71 last 12.59) - open/close 11988 (time 1777499356.61 total 679.56 last 12.45) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-12049) error: No space left on device total: 12049 open/close in 682.28 seconds: 17.66 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8986: kill: (112048) - No such process - unlinked 0 (time 1777499368 ; total 0 ; last 0) - unlinked 10000 (time 1777499432 ; total 64 ; last 64) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-12049) error: No such file or directory total: 12049 unlinks in 75 seconds: 160.653336 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg358-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg358-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg358-server unloading modules via unload_modules_local on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing unload_modules_local oleg358-server: modules unloaded. pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2