== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:58:01 (1784228281) start mds service on oleg453-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing load_modules_local oleg453-server: Loading modules from /home/green/git/lustre-release/lustre oleg453-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg453-server: detected 4 online CPUs by sysfs oleg453-server: MODOPTS_LIBCFS= oleg453-server: Force libcfs to create 2 CPU partitions oleg453-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg453-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg453-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg453-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg453-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-OST0000 oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 268 (time 1784228350.13 total 10.89 last 24.62) - open/close 442 (time 1784228360.13 total 20.89 last 17.39) - open/close 688 (time 1784228370.99 total 31.75 last 22.66) - open/close 865 (time 1784228381.24 total 41.99 last 17.28) - open/close 1034 (time 1784228391.28 total 52.03 last 16.83) - open/close 1166 (time 1784228402.09 total 62.85 last 12.20) - open/close 1469 (time 1784228412.10 total 72.86 last 30.27) - open/close 1647 (time 1784228422.44 total 83.20 last 17.21) - open/close 1779 (time 1784228433.13 total 93.89 last 12.35) - open/close 1874 (time 1784228443.18 total 103.94 last 9.45) - open/close 2003 (time 1784228453.93 total 114.69 last 12.00) - open/close 2137 (time 1784228464.75 total 125.51 last 12.39) - open/close 2463 (time 1784228474.93 total 135.68 last 32.04) - open/close 2655 (time 1784228485.04 total 145.80 last 18.99) - open/close 2940 (time 1784228495.86 total 156.62 last 26.33) - open/close 3268 (time 1784228506.87 total 167.63 last 29.79) - open/close 3583 (time 1784228517.29 total 178.05 last 30.24) - open/close 3798 (time 1784228528.11 total 188.87 last 19.87) - open/close 4024 (time 1784228540.91 total 201.67 last 17.65) - open/close 4268 (time 1784228550.92 total 211.68 last 24.38) - open/close 4588 (time 1784228561.59 total 222.35 last 30.00) - open/close 4957 (time 1784228572.59 total 233.34 last 33.56) - open/close 5167 (time 1784228583.28 total 244.04 last 19.64) - open/close 5445 (time 1784228593.97 total 254.72 last 26.01) - open/close 5774 (time 1784228603.97 total 264.73 last 32.89) - open/close 6054 (time 1784228614.89 total 275.65 last 25.63) - open/close 6296 (time 1784228625.71 total 286.47 last 22.36) - open/close 6602 (time 1784228635.71 total 296.47 last 30.60) - open/close 6880 (time 1784228645.93 total 306.69 last 27.21) - open/close 7123 (time 1784228655.98 total 316.74 last 24.19) - open/close 7392 (time 1784228665.98 total 326.74 last 26.89) - open/close 7710 (time 1784228676.40 total 337.15 last 30.54) - open/close 7974 (time 1784228687.42 total 348.17 last 23.96) - open/close 8318 (time 1784228697.42 total 358.18 last 34.39) - open/close 8572 (time 1784228707.42 total 368.18 last 25.39) - open/close 8825 (time 1784228717.42 total 378.18 last 25.29) - open/close 9154 (time 1784228728.11 total 388.86 last 30.80) - open/close 9452 (time 1784228738.48 total 399.24 last 28.72) - open/close 9766 (time 1784228749.17 total 409.92 last 29.39) - open/close 10000 (time 1784228759.14 total 419.90 last 23.45) - open/close 10146 (time 1784228769.32 total 430.08 last 14.34) - open/close 10447 (time 1784228779.64 total 440.40 last 29.19) - open/close 10775 (time 1784228789.64 total 450.40 last 32.80) - open/close 11159 (time 1784228799.64 total 460.40 last 38.39) - open/close 12171 (time 1784228809.64 total 470.40 last 101.19) - open/close 13910 (time 1784228819.65 total 480.40 last 173.84) - open/close 15629 (time 1784228829.65 total 490.41 last 171.82) - open/close 17465 (time 1784228839.66 total 500.41 last 183.51) - open/close 19246 (time 1784228849.66 total 510.41 last 178.10) - open/close 20000 (time 1784228854.18 total 514.93 last 166.79) - open/close 21475 (time 1784228864.19 total 524.94 last 147.36) - open/close 23216 (time 1784228874.19 total 534.95 last 174.06) - open/close 25519 (time 1784228884.19 total 544.95 last 230.23) - open/close 27819 (time 1784228894.19 total 554.95 last 229.96) - open/close 29571 (time 1784228904.20 total 564.95 last 175.14) - open/close 30000 (time 1784228907.20 total 567.96 last 142.92) - open/close 31526 (time 1784228917.21 total 577.96 last 152.48) - open/close 32896 (time 1784228927.21 total 587.97 last 136.91) - open/close 34582 (time 1784228937.22 total 597.98 last 168.51) - open/close 36137 (time 1784228947.22 total 607.98 last 155.42) - open/close 37966 (time 1784228957.22 total 617.98 last 182.88) - open/close 39792 (time 1784228967.23 total 627.98 last 182.55) - open/close 42305 (time 1784228977.23 total 637.99 last 251.23) - open/close 44269 (time 1784228987.23 total 647.99 last 196.35) - open/close 45947 (time 1784228997.23 total 657.99 last 167.79) - open/close 48126 (time 1784229007.24 total 667.99 last 217.82) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 676.38 seconds: 73.38 ops/second - unlinked 0 (time 1784229017 ; total 0 ; last 0) - unlinked 10000 (time 1784229053 ; total 36 ; last 36) - unlinked 20000 (time 1784229080 ; total 63 ; last 27) - unlinked 30000 (time 1784229109 ; total 92 ; last 29) - unlinked 40000 (time 1784229133 ; total 116 ; last 24) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 139 seconds: 357.057556 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg453-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg453-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg453-server unloading modules via unload_modules_local on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing unload_modules_local oleg453-server: modules unloaded. pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2