== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:24:53 (1782246293) Stopping clients: oleg435-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg435-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg435-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg435-server oleg435-server: oleg435-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg435-server: detected 4 online CPUs by sysfs oleg435-server: MODOPTS_LIBCFS= oleg435-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg435-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg435-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg435-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 654 (time 1782246500.03 total 10.02 last 65.24) - open/close 1304 (time 1782246510.03 total 20.03 last 64.96) - open/close 1981 (time 1782246520.05 total 30.04 last 67.62) - open/close 2578 (time 1782246530.08 total 40.08 last 59.47) - open/close 3192 (time 1782246540.08 total 50.08 last 61.40) - open/close 3789 (time 1782246550.11 total 60.10 last 59.58) - open/close 4391 (time 1782246560.11 total 70.11 last 60.15) total: 4500 open/close in 71.71 seconds: 62.75 ops/second - /unlink 1061 (time 1782246576.52 total 10.01 last 106.02) - /unlink 1835 (time 1782246586.52 total 20.01 last 77.36) - /unlink 2932 (time 1782246596.53 total 30.02 last 109.66) - /unlink 4124 (time 1782246606.53 total 40.02 last 119.15) total: 4500 /unlink in 43.28 seconds: 103.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 678 (time 1782246625.79 total 10.01 last 67.76) - open/close 1309 (time 1782246635.80 total 20.01 last 63.07) - open/close 1932 (time 1782246645.81 total 30.03 last 62.20) - open/close 2566 (time 1782246655.83 total 40.04 last 63.30) - open/close 3197 (time 1782246665.85 total 50.06 last 62.98) - open/close 3794 (time 1782246675.85 total 60.06 last 59.69) - open/close 4415 (time 1782246685.86 total 70.07 last 62.06) total: 4500 open/close in 71.77 seconds: 62.70 ops/second - /unlink 1073 (time 1782246703.27 total 10.00 last 107.26) - /unlink 2111 (time 1782246713.27 total 20.00 last 103.80) - /unlink 2932 (time 1782246723.28 total 30.01 last 82.04) - /unlink 3953 (time 1782246733.29 total 40.02 last 102.02) total: 4500 /unlink in 44.67 seconds: 100.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 556 (time 1782246752.43 total 10.01 last 55.54) - open/close 1041 (time 1782246762.44 total 20.02 last 48.45) - open/close 1576 (time 1782246772.46 total 30.04 last 53.38) - open/close 2145 (time 1782246782.48 total 40.06 last 56.80) - open/close 2779 (time 1782246792.49 total 50.07 last 63.32) - open/close 3421 (time 1782246802.51 total 60.09 last 64.10) - open/close 4112 (time 1782246812.54 total 70.12 last 68.92) total: 4500 open/close in 75.30 seconds: 59.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1209 (time 1782246832.99 total 10.01 last 120.75) - /unlink 2369 (time 1782246842.99 total 20.02 last 115.96) - /unlink 3416 (time 1782246853.01 total 30.03 last 104.52) - /unlink 4387 (time 1782246863.01 total 40.03 last 97.08) total: 4500 /unlink in 41.58 seconds: 108.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 520 (time 1782246879.82 total 10.02 last 51.92) - open/close 1032 (time 1782246889.83 total 20.02 last 51.16) - open/close 1433 (time 1782246899.84 total 30.03 last 40.06) - open/close 1867 (time 1782246909.85 total 40.04 last 43.37) - open/close 2514 (time 1782246919.85 total 50.04 last 64.69) - open/close 3127 (time 1782246929.85 total 60.05 last 61.27) - open/close 3701 (time 1782246939.87 total 70.06 last 57.32) - open/close 4239 (time 1782246949.87 total 80.07 last 53.78) total: 4500 open/close in 84.68 seconds: 53.14 ops/second - /unlink 1128 (time 1782246969.78 total 10.04 last 112.40) - /unlink 2191 (time 1782246979.78 total 20.04 last 106.29) - /unlink 3324 (time 1782246989.78 total 30.04 last 113.29) total: 4500 /unlink in 39.93 seconds: 112.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 591 (time 1782247014.80 total 10.00 last 59.08) - open/close 1210 (time 1782247024.81 total 20.01 last 61.84) - open/close 1849 (time 1782247034.82 total 30.02 last 63.82) - open/close 2457 (time 1782247044.82 total 40.03 last 60.79) - open/close 3127 (time 1782247054.84 total 50.05 last 66.87) - open/close 3763 (time 1782247064.85 total 60.05 last 63.59) - open/close 4352 (time 1782247074.85 total 70.06 last 58.85) total: 4500 open/close in 72.64 seconds: 61.95 ops/second - /unlink 1104 (time 1782247091.99 total 10.00 last 110.38) - /unlink 2331 (time 1782247102.00 total 20.01 last 122.56) - /unlink 3312 (time 1782247112.00 total 30.01 last 98.09) - /unlink 4366 (time 1782247122.00 total 40.02 last 105.35) total: 4500 /unlink in 41.67 seconds: 107.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 623 (time 1782247139.32 total 10.01 last 62.26) - open/close 1242 (time 1782247149.34 total 20.02 last 61.79) - open/close 1859 (time 1782247159.35 total 30.03 last 61.65) - open/close 2532 (time 1782247169.36 total 40.04 last 67.27) - open/close 3177 (time 1782247179.36 total 50.04 last 64.48) - open/close 3742 (time 1782247189.36 total 60.04 last 56.47) - open/close 4354 (time 1782247199.37 total 70.05 last 61.17) total: 4500 open/close in 72.21 seconds: 62.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1220 (time 1782247215.20 total 10.01 last 121.89) - /unlink 2359 (time 1782247225.20 total 20.01 last 113.86) - /unlink 3499 (time 1782247235.21 total 30.01 last 113.98) total: 4500 /unlink in 38.75 seconds: 116.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 627 (time 1782247259.18 total 10.00 last 62.68) - open/close 1306 (time 1782247269.18 total 20.01 last 67.86) - open/close 1980 (time 1782247279.19 total 30.02 last 67.34) - open/close 2652 (time 1782247289.19 total 40.02 last 67.18) - open/close 3268 (time 1782247299.20 total 50.03 last 61.54) - open/close 3911 (time 1782247309.21 total 60.04 last 64.25) total: 4500 open/close in 69.82 seconds: 64.45 ops/second - /unlink 1153 (time 1782247333.71 total 10.00 last 115.29) - /unlink 2302 (time 1782247343.72 total 20.00 last 114.90) - /unlink 3473 (time 1782247353.73 total 30.02 last 116.94) total: 4500 /unlink in 38.93 seconds: 115.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 633 (time 1782247378.07 total 10.00 last 63.28) - open/close 1197 (time 1782247388.08 total 20.02 last 56.33) - open/close 1793 (time 1782247398.09 total 30.03 last 59.54) - open/close 2398 (time 1782247408.09 total 40.03 last 60.46) - open/close 3025 (time 1782247418.10 total 50.04 last 62.64) - open/close 3609 (time 1782247428.11 total 60.05 last 58.35) - open/close 4143 (time 1782247438.12 total 70.06 last 53.38) total: 4500 open/close in 77.87 seconds: 57.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1147 (time 1782247464.31 total 10.00 last 114.69) - /unlink 2218 (time 1782247474.32 total 20.01 last 107.03) - /unlink 3336 (time 1782247484.33 total 30.02 last 111.65) - /unlink 4433 (time 1782247494.36 total 40.05 last 109.40) total: 4500 /unlink in 40.96 seconds: 109.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 607 (time 1782247510.90 total 10.00 last 60.68) - open/close 1274 (time 1782247520.90 total 20.01 last 66.68) - open/close 1900 (time 1782247530.92 total 30.03 last 62.47) - open/close 2526 (time 1782247540.92 total 40.03 last 62.58) - open/close 3190 (time 1782247550.93 total 50.04 last 66.36) - open/close 3824 (time 1782247560.94 total 60.05 last 63.34) - open/close 4472 (time 1782247570.95 total 70.05 last 64.75) total: 4500 open/close in 70.53 seconds: 63.81 ops/second - /unlink 1185 (time 1782247586.01 total 10.00 last 118.47) - /unlink 2413 (time 1782247596.02 total 20.01 last 122.65) - /unlink 3588 (time 1782247606.03 total 30.02 last 117.48) total: 4500 /unlink in 37.43 seconds: 120.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 593 (time 1782247627.51 total 10.01 last 59.23) - open/close 1219 (time 1782247637.52 total 20.02 last 62.54) - open/close 1850 (time 1782247647.53 total 30.03 last 63.04) - open/close 2460 (time 1782247657.53 total 40.03 last 61.00) - open/close 3093 (time 1782247667.54 total 50.04 last 63.23) - open/close 3616 (time 1782247677.57 total 60.07 last 52.16) - open/close 4176 (time 1782247687.58 total 70.08 last 55.92) total: 4500 open/close in 75.51 seconds: 59.60 ops/second - /unlink 1135 (time 1782247708.49 total 10.01 last 113.34) - /unlink 2302 (time 1782247718.50 total 20.02 last 116.64) - /unlink 3414 (time 1782247728.50 total 30.02 last 111.20) total: 4500 /unlink in 39.37 seconds: 114.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 662 (time 1782247753.38 total 10.02 last 66.06) - open/close 1312 (time 1782247763.40 total 20.04 last 64.88) - open/close 1939 (time 1782247773.41 total 30.05 last 62.66) - open/close 2536 (time 1782247783.41 total 40.05 last 59.70) - open/close 3136 (time 1782247793.42 total 50.06 last 59.93) - open/close 3735 (time 1782247803.42 total 60.06 last 59.89) - open/close 4284 (time 1782247813.43 total 70.07 last 54.83) total: 4500 open/close in 73.89 seconds: 60.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1131 (time 1782247832.25 total 10.00 last 113.06) - /unlink 2255 (time 1782247842.26 total 20.01 last 112.37) - /unlink 3375 (time 1782247852.26 total 30.01 last 111.97) total: 4500 /unlink in 39.95 seconds: 112.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 608 (time 1782247876.84 total 10.02 last 60.71) - open/close 1224 (time 1782247886.84 total 20.02 last 61.60) - open/close 1865 (time 1782247896.85 total 30.03 last 64.03) - open/close 2542 (time 1782247906.85 total 40.03 last 67.67) - open/close 3153 (time 1782247916.85 total 50.03 last 61.10) - open/close 3791 (time 1782247926.85 total 60.03 last 63.79) - open/close 4403 (time 1782247936.87 total 70.05 last 61.12) total: 4500 open/close in 71.82 seconds: 62.65 ops/second - /unlink 1007 (time 1782247954.06 total 10.01 last 100.59) - /unlink 2064 (time 1782247964.07 total 20.02 last 105.62) - /unlink 3166 (time 1782247974.09 total 30.04 last 110.01) - /unlink 4289 (time 1782247984.10 total 40.04 last 112.23) total: 4500 /unlink in 42.13 seconds: 106.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 546 (time 1782248001.46 total 10.02 last 54.50) - open/close 1107 (time 1782248011.48 total 20.03 last 56.01) - open/close 1733 (time 1782248021.48 total 30.04 last 62.55) - open/close 2378 (time 1782248031.49 total 40.05 last 64.46) - open/close 2974 (time 1782248041.50 total 50.06 last 59.53) - open/close 3568 (time 1782248051.52 total 60.07 last 59.32) - open/close 4182 (time 1782248061.52 total 70.08 last 61.39) total: 4500 open/close in 74.34 seconds: 60.53 ops/second - /unlink 1103 (time 1782248081.09 total 10.01 last 110.23) - /unlink 2177 (time 1782248091.10 total 20.01 last 107.35) - /unlink 3324 (time 1782248101.10 total 30.02 last 114.64) - /unlink 4370 (time 1782248111.11 total 40.02 last 104.55) total: 4500 /unlink in 40.98 seconds: 109.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 624 (time 1782248127.40 total 10.00 last 62.37) - open/close 1189 (time 1782248137.41 total 20.01 last 56.46) - open/close 1714 (time 1782248147.41 total 30.01 last 52.49) - open/close 2293 (time 1782248157.41 total 40.02 last 57.88) - open/close 2890 (time 1782248167.43 total 50.03 last 59.60) - open/close 3484 (time 1782248177.45 total 60.05 last 59.30) - open/close 4165 (time 1782248187.45 total 70.05 last 68.07) total: 4500 open/close in 76.39 seconds: 58.91 ops/second - /unlink 1028 (time 1782248209.33 total 10.00 last 102.78) - /unlink 2053 (time 1782248219.33 total 20.01 last 102.43) - /unlink 3230 (time 1782248229.33 total 30.01 last 117.69) - /unlink 4290 (time 1782248239.34 total 40.01 last 105.96) total: 4500 /unlink in 42.42 seconds: 106.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 567 (time 1782248257.59 total 10.01 last 56.67) - open/close 1158 (time 1782248267.60 total 20.01 last 59.05) - open/close 1807 (time 1782248277.61 total 30.02 last 64.84) - open/close 2381 (time 1782248287.63 total 40.04 last 57.28) - open/close 2954 (time 1782248297.63 total 50.05 last 57.29) - open/close 3584 (time 1782248307.64 total 60.05 last 62.99) - open/close 4163 (time 1782248317.65 total 70.06 last 57.84) total: 4500 open/close in 75.75 seconds: 59.41 ops/second - /unlink 1164 (time 1782248338.56 total 10.00 last 116.36) - /unlink 2182 (time 1782248348.56 total 20.01 last 101.73) - /unlink 3087 (time 1782248358.57 total 30.02 last 90.43) - /unlink 4192 (time 1782248368.58 total 40.02 last 110.45) total: 4500 /unlink in 42.56 seconds: 105.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 627 (time 1782248388.18 total 10.01 last 62.64) - open/close 1203 (time 1782248398.18 total 20.01 last 57.58) - open/close 1811 (time 1782248408.19 total 30.02 last 60.75) - open/close 2486 (time 1782248418.20 total 40.03 last 67.42) - open/close 3068 (time 1782248428.21 total 50.04 last 58.18) - open/close 3672 (time 1782248438.23 total 60.06 last 60.26) - open/close 4279 (time 1782248448.23 total 70.06 last 60.68) total: 4500 open/close in 73.55 seconds: 61.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1075 (time 1782248467.08 total 10.00 last 107.50) - /unlink 2254 (time 1782248477.09 total 20.01 last 117.75) - /unlink 3391 (time 1782248487.10 total 30.02 last 113.66) - /unlink 4462 (time 1782248497.11 total 40.03 last 106.93) total: 4500 /unlink in 40.41 seconds: 111.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 651 (time 1782248512.51 total 10.00 last 65.08) - open/close 1234 (time 1782248522.51 total 20.00 last 58.30) - open/close 1815 (time 1782248532.53 total 30.02 last 58.03) - open/close 2398 (time 1782248542.54 total 40.03 last 58.22) - open/close 3015 (time 1782248552.54 total 50.03 last 61.67) - open/close 3637 (time 1782248562.58 total 60.07 last 62.00) - open/close 4261 (time 1782248572.58 total 70.07 last 62.36) total: 4500 open/close in 74.08 seconds: 60.75 ops/second - /unlink 1091 (time 1782248591.77 total 10.00 last 109.08) - /unlink 2256 (time 1782248601.78 total 20.01 last 116.44) - /unlink 3275 (time 1782248611.79 total 30.02 last 101.78) - /unlink 4400 (time 1782248621.80 total 40.03 last 112.42) total: 4500 /unlink in 41.05 seconds: 109.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 666 (time 1782248639.24 total 10.01 last 66.56) - open/close 1228 (time 1782248649.25 total 20.02 last 56.14) - open/close 1770 (time 1782248659.25 total 30.02 last 54.17) - open/close 2367 (time 1782248669.26 total 40.03 last 59.63) - open/close 2984 (time 1782248679.28 total 50.05 last 61.61) - open/close 3606 (time 1782248689.28 total 60.05 last 62.16) - open/close 4249 (time 1782248699.29 total 70.06 last 64.28) total: 4500 open/close in 73.62 seconds: 61.13 ops/second - /unlink 1135 (time 1782248716.73 total 10.01 last 113.41) - /unlink 2139 (time 1782248726.73 total 20.01 last 100.37) - /unlink 3282 (time 1782248736.75 total 30.03 last 114.09) - /unlink 4245 (time 1782248746.76 total 40.04 last 96.22) total: 4500 /unlink in 42.38 seconds: 106.17 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 94974 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg435-server unloading modules via unload_modules_local on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing unload_modules_local oleg435-server: modules unloaded.