== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:53:41 (1780325621) start mds service on oleg414-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg414-server: detected 4 online CPUs by sysfs oleg414-server: MODOPTS_LIBCFS= oleg414-server: Force libcfs to create 2 CPU partitions oleg414-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg414-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg414-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg414-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 259 (time 1780325663.07 total 10.09 last 25.66) - open/close 469 (time 1780325673.69 total 20.72 last 19.77) - open/close 680 (time 1780325684.13 total 31.15 last 20.22) - open/close 900 (time 1780325695.14 total 42.16 last 19.98) - open/close 1189 (time 1780325705.51 total 52.53 last 27.87) - open/close 1415 (time 1780325715.51 total 62.54 last 22.59) - open/close 1728 (time 1780325726.24 total 73.26 last 29.18) - open/close 2062 (time 1780325736.24 total 83.26 last 33.39) - open/close 2324 (time 1780325746.24 total 93.27 last 26.20) - open/close 2626 (time 1780325756.70 total 103.73 last 28.86) - open/close 2913 (time 1780325767.21 total 114.23 last 27.33) - open/close 3221 (time 1780325777.83 total 124.86 last 28.98) - open/close 3546 (time 1780325788.45 total 135.47 last 30.62) - open/close 3893 (time 1780325798.45 total 145.47 last 34.69) - open/close 4209 (time 1780325808.45 total 155.48 last 31.58) - open/close 4519 (time 1780325818.85 total 165.88 last 29.80) - open/close 4740 (time 1780325828.86 total 175.88 last 22.09) - open/close 5054 (time 1780325839.39 total 186.41 last 29.82) - open/close 5290 (time 1780325849.39 total 196.41 last 23.60) - open/close 5501 (time 1780325860.00 total 207.02 last 19.89) - open/close 5637 (time 1780325872.10 total 219.12 last 11.24) - open/close 5842 (time 1780325882.72 total 229.74 last 19.31) - open/close 6070 (time 1780325893.53 total 240.56 last 21.08) - open/close 6302 (time 1780325904.29 total 251.31 last 21.58) - open/close 6541 (time 1780325915.04 total 262.07 last 22.21) - open/close 6842 (time 1780325925.73 total 272.75 last 28.18) - open/close 7064 (time 1780325936.04 total 283.06 last 21.53) - open/close 7267 (time 1780325946.53 total 293.55 last 19.35) - open/close 7670 (time 1780325956.53 total 303.55 last 40.30) - open/close 7860 (time 1780325966.69 total 313.71 last 18.71) - open/close 8152 (time 1780325977.18 total 324.20 last 27.82) - open/close 8843 (time 1780325987.55 total 334.57 last 66.64) - open/close 9468 (time 1780325997.55 total 344.57 last 62.49) - open/close 10000 (time 1780326005.35 total 352.38 last 68.18) - open/close 10439 (time 1780326015.35 total 362.38 last 43.90) - open/close 11034 (time 1780326025.82 total 372.84 last 56.86) - open/close 11724 (time 1780326035.82 total 382.84 last 69.00) - open/close 12421 (time 1780326046.05 total 393.07 last 68.16) - open/close 16100 (time 1780326056.05 total 403.07 last 367.86) - open/close 19725 (time 1780326066.05 total 413.07 last 362.47) - open/close 22866 (time 1780326076.05 total 423.07 last 314.04) - open/close 26243 (time 1780326086.05 total 433.07 last 337.67) - open/close 30000 (time 1780326095.28 total 442.31 last 406.87) - open/close 34311 (time 1780326105.28 total 452.31 last 431.08) - open/close 37639 (time 1780326115.29 total 462.31 last 332.74) - open/close 40000 (time 1780326122.33 total 469.35 last 335.40) - open/close 42969 (time 1780326132.33 total 479.35 last 296.84) - open/close 46187 (time 1780326142.33 total 489.35 last 321.72) - open/close 49560 (time 1780326152.33 total 499.36 last 337.21) total: 50000 open/close in 500.81 seconds: 99.84 ops/second - unlinked 0 (time 1780326155 ; total 0 ; last 0) - unlinked 10000 (time 1780326173 ; total 18 ; last 18) - unlinked 20000 (time 1780326188 ; total 33 ; last 15) - unlinked 30000 (time 1780326207 ; total 52 ; last 19) - unlinked 40000 (time 1780326228 ; total 73 ; last 21) total: 50000 unlinks in 92 seconds: 543.478271 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server unloading modules via unload_modules_local on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing unload_modules_local oleg414-server: modules unloaded. pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2