== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:11:08 (1788441068) Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg408-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg408-server oleg408-server: oleg408-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: kmemleak disabled oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 523 (time 1788441294.76 total 10.02 last 52.20) - open/close 990 (time 1788441304.78 total 20.04 last 46.61) - open/close 1604 (time 1788441314.79 total 30.05 last 61.34) - open/close 2229 (time 1788441324.81 total 40.07 last 62.37) - open/close 2874 (time 1788441334.82 total 50.09 last 64.39) - open/close 3458 (time 1788441344.84 total 60.10 last 58.33) - open/close 4010 (time 1788441354.84 total 70.11 last 55.17) total: 4500 open/close in 79.85 seconds: 56.35 ops/second - /unlink 1150 (time 1788441380.27 total 10.00 last 114.97) - /unlink 2371 (time 1788441390.27 total 20.00 last 122.09) - /unlink 3583 (time 1788441400.27 total 30.00 last 121.19) total: 4500 /unlink in 38.36 seconds: 117.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 596 (time 1788441424.11 total 10.01 last 59.54) - open/close 1216 (time 1788441434.13 total 20.02 last 61.93) - open/close 1710 (time 1788441444.15 total 30.05 last 49.28) - open/close 2331 (time 1788441454.17 total 40.07 last 61.97) - open/close 2926 (time 1788441464.19 total 50.09 last 59.38) - open/close 3567 (time 1788441474.19 total 60.09 last 64.08) - open/close 4183 (time 1788441484.20 total 70.09 last 61.58) total: 4500 open/close in 75.68 seconds: 59.46 ops/second - /unlink 1183 (time 1788441505.45 total 10.00 last 118.26) - /unlink 2320 (time 1788441515.45 total 20.00 last 113.70) - /unlink 3462 (time 1788441525.47 total 30.03 last 113.93) total: 4500 /unlink in 39.20 seconds: 114.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 665 (time 1788441549.85 total 10.02 last 66.37) - open/close 1269 (time 1788441559.85 total 20.03 last 60.36) - open/close 1908 (time 1788441569.87 total 30.05 last 63.76) - open/close 2502 (time 1788441579.88 total 40.05 last 59.39) - open/close 3099 (time 1788441589.88 total 50.05 last 59.69) - open/close 3703 (time 1788441599.89 total 60.06 last 60.34) - open/close 4268 (time 1788441609.90 total 70.07 last 56.45) total: 4500 open/close in 73.95 seconds: 60.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1250 (time 1788441629.04 total 10.00 last 124.95) - /unlink 2434 (time 1788441639.05 total 20.02 last 118.26) - /unlink 3550 (time 1788441649.05 total 30.02 last 111.58) total: 4500 /unlink in 38.22 seconds: 117.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 632 (time 1788441672.82 total 10.01 last 63.14) - open/close 1239 (time 1788441682.84 total 20.02 last 60.61) - open/close 1905 (time 1788441692.85 total 30.04 last 66.51) - open/close 2538 (time 1788441702.86 total 40.04 last 63.25) - open/close 3196 (time 1788441712.87 total 50.06 last 65.70) - open/close 3750 (time 1788441722.87 total 60.06 last 55.38) - open/close 4271 (time 1788441732.90 total 70.09 last 51.95) total: 4500 open/close in 73.54 seconds: 61.19 ops/second - /unlink 1161 (time 1788441751.81 total 10.02 last 115.84) - /unlink 2157 (time 1788441761.81 total 20.03 last 99.50) - /unlink 3199 (time 1788441771.83 total 30.05 last 104.05) - /unlink 4169 (time 1788441781.84 total 40.05 last 96.94) total: 4500 /unlink in 43.08 seconds: 104.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 675 (time 1788441799.64 total 10.00 last 67.47) - open/close 1384 (time 1788441809.65 total 20.02 last 70.80) - open/close 2083 (time 1788441819.67 total 30.04 last 69.78) - open/close 2767 (time 1788441829.67 total 40.04 last 68.38) - open/close 3367 (time 1788441839.67 total 50.04 last 60.00) - open/close 4027 (time 1788441849.67 total 60.04 last 65.97) total: 4500 open/close in 67.15 seconds: 67.01 ops/second - /unlink 1142 (time 1788441871.24 total 10.01 last 114.05) - /unlink 2371 (time 1788441881.24 total 20.02 last 122.86) - /unlink 3548 (time 1788441891.25 total 30.02 last 117.64) total: 4500 /unlink in 38.05 seconds: 118.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 575 (time 1788441914.81 total 10.01 last 57.45) - open/close 1212 (time 1788441924.82 total 20.02 last 63.66) - open/close 1856 (time 1788441934.85 total 30.05 last 64.21) - open/close 2480 (time 1788441944.86 total 40.05 last 62.36) - open/close 3204 (time 1788441954.86 total 50.06 last 72.33) - open/close 3767 (time 1788441964.87 total 60.07 last 56.24) - open/close 4327 (time 1788441974.89 total 70.08 last 55.94) total: 4500 open/close in 72.67 seconds: 61.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1178 (time 1788441992.66 total 10.00 last 117.79) - /unlink 2407 (time 1788442002.66 total 20.00 last 122.85) - /unlink 3606 (time 1788442012.68 total 30.02 last 119.73) total: 4500 /unlink in 37.95 seconds: 118.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 624 (time 1788442036.00 total 10.01 last 62.36) - open/close 1247 (time 1788442046.03 total 20.04 last 62.08) - open/close 1811 (time 1788442056.05 total 30.06 last 56.29) - open/close 2409 (time 1788442066.06 total 40.07 last 59.76) - open/close 2984 (time 1788442076.06 total 50.07 last 57.46) - open/close 3579 (time 1788442086.10 total 60.10 last 59.31) - open/close 4126 (time 1788442096.11 total 70.12 last 54.62) total: 4500 open/close in 75.67 seconds: 59.47 ops/second - /unlink 1235 (time 1788442116.97 total 10.00 last 123.49) - /unlink 2274 (time 1788442126.97 total 20.01 last 103.85) - /unlink 3335 (time 1788442136.98 total 30.01 last 106.02) - /unlink 4324 (time 1788442146.98 total 40.02 last 98.87) total: 4500 /unlink in 41.56 seconds: 108.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 593 (time 1788442163.70 total 10.00 last 59.28) - open/close 1194 (time 1788442173.70 total 20.01 last 60.07) - open/close 1802 (time 1788442183.71 total 30.01 last 60.76) - open/close 2378 (time 1788442193.71 total 40.02 last 57.58) - open/close 3027 (time 1788442203.73 total 50.03 last 64.80) - open/close 3625 (time 1788442213.74 total 60.04 last 59.74) - open/close 4206 (time 1788442223.75 total 70.06 last 58.02) total: 4500 open/close in 74.99 seconds: 60.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1188 (time 1788442243.77 total 10.01 last 118.70) - /unlink 2252 (time 1788442253.77 total 20.01 last 106.36) - /unlink 3453 (time 1788442263.78 total 30.02 last 119.96) total: 4500 /unlink in 38.80 seconds: 115.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 605 (time 1788442287.99 total 10.00 last 60.49) - open/close 1204 (time 1788442298.00 total 20.02 last 59.80) - open/close 1830 (time 1788442308.01 total 30.03 last 62.55) - open/close 2459 (time 1788442318.03 total 40.05 last 62.76) - open/close 3059 (time 1788442328.04 total 50.05 last 59.99) - open/close 3651 (time 1788442338.04 total 60.05 last 59.19) - open/close 4162 (time 1788442348.06 total 70.07 last 51.00) total: 4500 open/close in 76.56 seconds: 58.78 ops/second - /unlink 1092 (time 1788442370.93 total 10.00 last 109.15) - /unlink 2208 (time 1788442380.93 total 20.01 last 111.56) - /unlink 3333 (time 1788442390.95 total 30.02 last 112.37) - /unlink 4381 (time 1788442400.95 total 40.02 last 104.75) total: 4500 /unlink in 41.60 seconds: 108.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 603 (time 1788442418.58 total 10.00 last 60.29) - open/close 1175 (time 1788442428.61 total 20.03 last 57.03) - open/close 1835 (time 1788442438.61 total 30.04 last 65.96) - open/close 2429 (time 1788442448.62 total 40.04 last 59.36) - open/close 3085 (time 1788442458.62 total 50.05 last 65.57) - open/close 3763 (time 1788442468.63 total 60.05 last 67.78) - open/close 4456 (time 1788442478.64 total 70.06 last 69.23) total: 4500 open/close in 70.64 seconds: 63.70 ops/second - /unlink 1091 (time 1788442494.62 total 10.01 last 109.04) - /unlink 2212 (time 1788442504.62 total 20.01 last 112.09) - /unlink 3413 (time 1788442514.63 total 30.02 last 119.97) total: 4500 /unlink in 39.28 seconds: 114.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 586 (time 1788442538.55 total 10.01 last 58.53) - open/close 1132 (time 1788442548.55 total 20.02 last 54.57) - open/close 1829 (time 1788442558.56 total 30.02 last 69.66) - open/close 2448 (time 1788442568.57 total 40.03 last 61.85) - open/close 3121 (time 1788442578.58 total 50.04 last 67.25) - open/close 3800 (time 1788442588.58 total 60.04 last 67.86) - open/close 4415 (time 1788442598.58 total 70.04 last 61.50) total: 4500 open/close in 71.32 seconds: 63.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1215 (time 1788442615.01 total 10.00 last 121.47) - /unlink 2305 (time 1788442625.02 total 20.01 last 108.95) - /unlink 3574 (time 1788442635.02 total 30.01 last 126.86) total: 4500 /unlink in 37.86 seconds: 118.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 642 (time 1788442658.10 total 10.00 last 64.19) - open/close 1257 (time 1788442668.11 total 20.01 last 61.45) - open/close 1865 (time 1788442678.13 total 30.03 last 60.69) - open/close 2490 (time 1788442688.13 total 40.03 last 62.47) - open/close 3175 (time 1788442698.15 total 50.05 last 68.36) - open/close 3776 (time 1788442708.16 total 60.06 last 60.05) - open/close 4380 (time 1788442718.17 total 70.07 last 60.35) total: 4500 open/close in 71.97 seconds: 62.53 ops/second - /unlink 1244 (time 1788442734.92 total 10.00 last 124.39) - /unlink 2509 (time 1788442744.93 total 20.01 last 126.44) - /unlink 3682 (time 1788442754.93 total 30.01 last 117.27) total: 4500 /unlink in 37.19 seconds: 120.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 620 (time 1788442777.15 total 10.01 last 61.94) - open/close 1267 (time 1788442787.16 total 20.02 last 64.66) - open/close 1874 (time 1788442797.17 total 30.02 last 60.65) - open/close 2451 (time 1788442807.18 total 40.03 last 57.64) - open/close 3063 (time 1788442817.18 total 50.04 last 61.17) - open/close 3669 (time 1788442827.20 total 60.06 last 60.50) - open/close 4251 (time 1788442837.23 total 70.09 last 58.00) total: 4500 open/close in 74.58 seconds: 60.34 ops/second - /unlink 1213 (time 1788442857.34 total 10.01 last 121.22) - /unlink 2488 (time 1788442867.34 total 20.01 last 127.40) - /unlink 3689 (time 1788442877.35 total 30.02 last 120.04) total: 4500 /unlink in 36.88 seconds: 122.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 644 (time 1788442899.47 total 10.01 last 64.32) - open/close 1227 (time 1788442909.48 total 20.02 last 58.23) - open/close 1919 (time 1788442919.49 total 30.03 last 69.14) - open/close 2440 (time 1788442929.49 total 40.04 last 52.08) - open/close 2953 (time 1788442939.49 total 50.04 last 51.28) - open/close 3532 (time 1788442949.50 total 60.04 last 57.88) - open/close 4148 (time 1788442959.50 total 70.04 last 61.58) total: 4500 open/close in 75.92 seconds: 59.27 ops/second - /unlink 1237 (time 1788442980.21 total 10.01 last 123.63) - /unlink 2374 (time 1788442990.21 total 20.01 last 113.69) - /unlink 3550 (time 1788443000.21 total 30.01 last 117.55) total: 4500 /unlink in 38.24 seconds: 117.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 628 (time 1788443024.67 total 10.00 last 62.77) - open/close 1284 (time 1788443034.68 total 20.01 last 65.55) - open/close 1951 (time 1788443044.68 total 30.02 last 66.66) - open/close 2570 (time 1788443054.69 total 40.03 last 61.83) - open/close 3218 (time 1788443064.70 total 50.04 last 64.74) - open/close 3853 (time 1788443074.71 total 60.04 last 63.47) - open/close 4457 (time 1788443084.72 total 70.05 last 60.34) total: 4500 open/close in 70.65 seconds: 63.69 ops/second - /unlink 1102 (time 1788443101.16 total 10.00 last 110.16) - /unlink 2264 (time 1788443111.16 total 20.01 last 116.17) - /unlink 3353 (time 1788443121.16 total 30.01 last 108.88) total: 4500 /unlink in 39.92 seconds: 112.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 564 (time 1788443147.06 total 10.03 last 56.24) - open/close 1174 (time 1788443157.06 total 20.03 last 61.00) - open/close 1747 (time 1788443167.06 total 30.03 last 57.29) - open/close 2301 (time 1788443177.06 total 40.03 last 55.40) - open/close 2915 (time 1788443187.09 total 50.06 last 61.22) - open/close 3550 (time 1788443197.09 total 60.06 last 63.50) - open/close 4149 (time 1788443207.10 total 70.06 last 59.89) total: 4500 open/close in 76.10 seconds: 59.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1062 (time 1788443228.78 total 10.02 last 106.04) - /unlink 2085 (time 1788443238.79 total 20.03 last 102.17) - /unlink 3124 (time 1788443248.80 total 30.03 last 103.88) - /unlink 4183 (time 1788443258.80 total 40.03 last 105.89) total: 4500 /unlink in 43.86 seconds: 102.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 555 (time 1788443279.76 total 10.01 last 55.46) - open/close 1103 (time 1788443289.77 total 20.02 last 54.75) - open/close 1730 (time 1788443299.78 total 30.02 last 62.68) - open/close 2223 (time 1788443309.79 total 40.03 last 49.24) - open/close 2757 (time 1788443319.79 total 50.03 last 53.39) - open/close 3248 (time 1788443329.81 total 60.05 last 49.01) - open/close 3779 (time 1788443339.81 total 70.05 last 53.10) - open/close 4281 (time 1788443349.82 total 80.06 last 50.15) total: 4500 open/close in 84.07 seconds: 53.53 ops/second - /unlink 1259 (time 1788443368.62 total 10.00 last 125.88) - /unlink 2424 (time 1788443378.62 total 20.00 last 116.48) - /unlink 3639 (time 1788443388.64 total 30.02 last 121.27) total: 4500 /unlink in 37.38 seconds: 120.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 633 (time 1788443411.46 total 10.01 last 63.24) - open/close 1255 (time 1788443421.47 total 20.02 last 62.14) - open/close 1772 (time 1788443431.50 total 30.05 last 51.54) - open/close 2367 (time 1788443441.50 total 40.05 last 59.49) - open/close 2845 (time 1788443451.50 total 50.05 last 47.79) - open/close 3425 (time 1788443461.52 total 60.07 last 57.91) - open/close 4009 (time 1788443471.53 total 70.08 last 58.36) total: 4500 open/close in 77.48 seconds: 58.08 ops/second - /unlink 1223 (time 1788443493.87 total 10.01 last 122.22) - /unlink 2347 (time 1788443503.87 total 20.01 last 112.37) - /unlink 3560 (time 1788443513.88 total 30.02 last 121.21) total: 4500 /unlink in 38.27 seconds: 117.57 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95369 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: kmemleak disabled modules unloaded.