== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:12:58 (1787872378) Stopping clients: oleg360-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg360-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg360-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg360-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg360-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg360-server oleg360-server: oleg360-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing load_modules_local oleg360-server: Loading modules from /home/green/git/lustre-release/lustre oleg360-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg360-server: detected 4 online CPUs by sysfs oleg360-server: MODOPTS_LIBCFS= oleg360-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg360-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg360-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg360-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 591 (time 1787872597.02 total 10.01 last 59.07) - open/close 1254 (time 1787872607.02 total 20.01 last 66.27) - open/close 1871 (time 1787872617.03 total 30.02 last 61.67) - open/close 2521 (time 1787872627.06 total 40.05 last 64.78) - open/close 3124 (time 1787872637.07 total 50.06 last 60.24) - open/close 3722 (time 1787872647.07 total 60.06 last 59.80) - open/close 4379 (time 1787872657.09 total 70.08 last 65.59) total: 4500 open/close in 71.93 seconds: 62.56 ops/second - /unlink 1204 (time 1787872674.24 total 10.00 last 120.39) - /unlink 2461 (time 1787872684.25 total 20.01 last 125.56) - /unlink 3566 (time 1787872694.27 total 30.03 last 110.32) total: 4500 /unlink in 37.15 seconds: 121.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 591 (time 1787872717.26 total 10.01 last 59.06) - open/close 1162 (time 1787872727.26 total 20.01 last 57.07) - open/close 1818 (time 1787872737.27 total 30.02 last 65.54) - open/close 2501 (time 1787872747.27 total 40.02 last 68.29) - open/close 3180 (time 1787872757.28 total 50.03 last 67.83) - open/close 3893 (time 1787872767.29 total 60.04 last 71.23) total: 4500 open/close in 69.74 seconds: 64.53 ops/second - /unlink 1275 (time 1787872792.10 total 10.00 last 127.47) - /unlink 2539 (time 1787872802.10 total 20.01 last 126.35) - /unlink 3840 (time 1787872812.11 total 30.01 last 130.02) total: 4500 /unlink in 35.03 seconds: 128.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 607 (time 1787872831.94 total 10.02 last 60.59) - open/close 1208 (time 1787872841.97 total 20.04 last 59.97) - open/close 1888 (time 1787872851.97 total 30.04 last 67.98) - open/close 2543 (time 1787872861.98 total 40.05 last 65.43) - open/close 3151 (time 1787872872.00 total 50.07 last 60.70) - open/close 3875 (time 1787872882.00 total 60.08 last 72.34) total: 4500 open/close in 69.49 seconds: 64.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1313 (time 1787872906.37 total 10.00 last 131.24) - /unlink 2578 (time 1787872916.38 total 20.01 last 126.37) - /unlink 3880 (time 1787872926.38 total 30.02 last 130.19) total: 4500 /unlink in 34.76 seconds: 129.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 699 (time 1787872945.50 total 10.01 last 69.83) - open/close 1422 (time 1787872955.51 total 20.02 last 72.23) - open/close 2136 (time 1787872965.51 total 30.02 last 71.40) - open/close 2834 (time 1787872975.51 total 40.02 last 69.79) - open/close 3513 (time 1787872985.51 total 50.03 last 67.85) - open/close 4206 (time 1787872995.52 total 60.03 last 69.28) total: 4500 open/close in 64.09 seconds: 70.22 ops/second - /unlink 1346 (time 1787873013.72 total 10.01 last 134.53) - /unlink 2669 (time 1787873023.72 total 20.01 last 132.28) - /unlink 3882 (time 1787873033.72 total 30.01 last 121.24) total: 4500 /unlink in 35.04 seconds: 128.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 652 (time 1787873053.77 total 10.01 last 65.16) - open/close 1236 (time 1787873063.77 total 20.01 last 58.35) - open/close 1889 (time 1787873073.78 total 30.02 last 65.28) - open/close 2576 (time 1787873083.78 total 40.02 last 68.70) - open/close 3240 (time 1787873093.79 total 50.03 last 66.31) - open/close 3909 (time 1787873103.79 total 60.03 last 66.88) total: 4500 open/close in 69.20 seconds: 65.03 ops/second - /unlink 1329 (time 1787873127.07 total 10.01 last 132.83) - /unlink 2681 (time 1787873137.07 total 20.01 last 135.17) - /unlink 3910 (time 1787873147.07 total 30.01 last 122.88) total: 4500 /unlink in 34.34 seconds: 131.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 703 (time 1787873166.43 total 10.00 last 70.29) - open/close 1417 (time 1787873176.45 total 20.01 last 71.31) - open/close 2062 (time 1787873186.45 total 30.02 last 64.49) - open/close 2670 (time 1787873196.46 total 40.02 last 60.76) - open/close 3248 (time 1787873206.46 total 50.03 last 57.76) - open/close 3830 (time 1787873216.47 total 60.04 last 58.16) - open/close 4458 (time 1787873226.48 total 70.05 last 62.74) total: 4500 open/close in 70.59 seconds: 63.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1152 (time 1787873241.70 total 10.00 last 115.16) - /unlink 2381 (time 1787873251.71 total 20.01 last 122.81) - /unlink 3632 (time 1787873261.71 total 30.01 last 125.07) total: 4500 /unlink in 37.03 seconds: 121.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 658 (time 1787873283.65 total 10.02 last 65.70) - open/close 1382 (time 1787873293.66 total 20.02 last 72.35) - open/close 2068 (time 1787873303.66 total 30.02 last 68.58) - open/close 2765 (time 1787873313.69 total 40.06 last 69.47) - open/close 3422 (time 1787873323.69 total 50.06 last 65.69) - open/close 4064 (time 1787873333.69 total 60.06 last 64.19) total: 4500 open/close in 66.79 seconds: 67.38 ops/second - /unlink 1157 (time 1787873355.22 total 10.00 last 115.66) - /unlink 2445 (time 1787873365.22 total 20.01 last 128.77) - /unlink 3328 (time 1787873375.23 total 30.01 last 88.29) - /unlink 4268 (time 1787873385.23 total 40.01 last 93.98) total: 4500 /unlink in 43.14 seconds: 104.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 642 (time 1787873403.51 total 10.02 last 64.04) - open/close 1352 (time 1787873413.51 total 20.03 last 70.98) - open/close 2037 (time 1787873423.51 total 30.03 last 68.47) - open/close 2722 (time 1787873433.52 total 40.04 last 68.42) - open/close 3412 (time 1787873443.54 total 50.06 last 68.91) - open/close 4060 (time 1787873453.55 total 60.07 last 64.73) total: 4500 open/close in 66.38 seconds: 67.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1285 (time 1787873475.05 total 10.01 last 128.42) - /unlink 2561 (time 1787873485.06 total 20.01 last 127.55) - /unlink 3758 (time 1787873495.06 total 30.02 last 119.63) total: 4500 /unlink in 37.06 seconds: 121.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 634 (time 1787873516.88 total 10.01 last 63.33) - open/close 1337 (time 1787873526.88 total 20.01 last 70.28) - open/close 2034 (time 1787873536.89 total 30.02 last 69.64) - open/close 2692 (time 1787873546.90 total 40.03 last 65.73) - open/close 3446 (time 1787873556.91 total 50.04 last 75.36) - open/close 4111 (time 1787873566.91 total 60.04 last 66.47) total: 4500 open/close in 65.60 seconds: 68.60 ops/second - /unlink 1278 (time 1787873587.46 total 10.01 last 127.70) - /unlink 2520 (time 1787873597.47 total 20.02 last 124.08) - /unlink 3763 (time 1787873607.47 total 30.02 last 124.27) total: 4500 /unlink in 36.24 seconds: 124.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 672 (time 1787873628.86 total 10.02 last 67.05) - open/close 1318 (time 1787873638.86 total 20.03 last 64.57) - open/close 2002 (time 1787873648.87 total 30.04 last 68.35) - open/close 2705 (time 1787873658.88 total 40.05 last 70.21) - open/close 3325 (time 1787873668.88 total 50.05 last 61.99) - open/close 4015 (time 1787873678.89 total 60.05 last 68.97) total: 4500 open/close in 66.88 seconds: 67.28 ops/second - /unlink 1308 (time 1787873700.13 total 10.00 last 130.77) - /unlink 2480 (time 1787873710.13 total 20.00 last 117.19) - /unlink 3718 (time 1787873720.14 total 30.01 last 123.73) total: 4500 /unlink in 36.99 seconds: 121.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 694 (time 1787873741.73 total 10.01 last 69.33) - open/close 1318 (time 1787873751.74 total 20.01 last 62.38) - open/close 2004 (time 1787873761.75 total 30.02 last 68.53) - open/close 2716 (time 1787873771.75 total 40.03 last 71.17) - open/close 3380 (time 1787873781.77 total 50.05 last 66.26) - open/close 4101 (time 1787873791.78 total 60.06 last 72.01) total: 4500 open/close in 65.69 seconds: 68.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1263 (time 1787873811.84 total 10.01 last 126.17) - /unlink 2377 (time 1787873821.85 total 20.02 last 111.25) - /unlink 3626 (time 1787873831.86 total 30.03 last 124.81) total: 4500 /unlink in 37.42 seconds: 120.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 610 (time 1787873854.31 total 10.01 last 60.91) - open/close 1246 (time 1787873864.33 total 20.03 last 63.49) - open/close 1929 (time 1787873874.34 total 30.04 last 68.24) - open/close 2571 (time 1787873884.34 total 40.05 last 64.18) - open/close 3207 (time 1787873894.35 total 50.06 last 63.51) - open/close 3824 (time 1787873904.37 total 60.08 last 61.58) - open/close 4455 (time 1787873914.38 total 70.09 last 63.04) total: 4500 open/close in 70.66 seconds: 63.69 ops/second - /unlink 1251 (time 1787873929.42 total 10.00 last 125.06) - /unlink 2454 (time 1787873939.42 total 20.01 last 120.28) - /unlink 3744 (time 1787873949.42 total 30.01 last 128.97) total: 4500 /unlink in 35.98 seconds: 125.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 671 (time 1787873969.90 total 10.00 last 67.08) - open/close 1321 (time 1787873979.90 total 20.00 last 64.99) - open/close 1947 (time 1787873989.91 total 30.01 last 62.56) - open/close 2613 (time 1787873999.92 total 40.02 last 66.50) - open/close 3363 (time 1787874009.93 total 50.03 last 74.96) - open/close 4063 (time 1787874019.93 total 60.04 last 69.95) total: 4500 open/close in 66.20 seconds: 67.98 ops/second - /unlink 1282 (time 1787874041.45 total 10.00 last 128.16) - /unlink 2560 (time 1787874051.45 total 20.01 last 127.76) - /unlink 3851 (time 1787874061.46 total 30.01 last 129.04) total: 4500 /unlink in 35.54 seconds: 126.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 718 (time 1787874081.63 total 10.01 last 71.72) - open/close 1406 (time 1787874091.64 total 20.02 last 68.75) - open/close 2162 (time 1787874101.65 total 30.03 last 75.48) - open/close 2851 (time 1787874111.66 total 40.04 last 68.86) - open/close 3577 (time 1787874121.67 total 50.05 last 72.52) - open/close 4255 (time 1787874131.67 total 60.05 last 67.77) total: 4500 open/close in 63.97 seconds: 70.35 ops/second - /unlink 1225 (time 1787874149.77 total 10.01 last 122.41) - /unlink 2447 (time 1787874159.77 total 20.01 last 122.14) - /unlink 3701 (time 1787874169.78 total 30.02 last 125.31) total: 4500 /unlink in 36.58 seconds: 123.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 704 (time 1787874191.08 total 10.01 last 70.35) - open/close 1418 (time 1787874201.09 total 20.02 last 71.33) - open/close 2136 (time 1787874211.10 total 30.03 last 71.72) - open/close 2851 (time 1787874221.11 total 40.04 last 71.45) - open/close 3536 (time 1787874231.12 total 50.04 last 68.44) - open/close 4122 (time 1787874241.14 total 60.07 last 58.45) total: 4500 open/close in 65.32 seconds: 68.89 ops/second - /unlink 1264 (time 1787874261.04 total 10.01 last 126.31) - /unlink 2439 (time 1787874271.04 total 20.01 last 117.50) - /unlink 3673 (time 1787874281.06 total 30.02 last 123.22) total: 4500 /unlink in 36.95 seconds: 121.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 712 (time 1787874303.72 total 10.00 last 71.17) - open/close 1272 (time 1787874313.73 total 20.01 last 55.95) - open/close 1952 (time 1787874323.73 total 30.02 last 67.95) - open/close 2637 (time 1787874333.75 total 40.03 last 68.40) - open/close 3319 (time 1787874343.75 total 50.03 last 68.20) - open/close 4077 (time 1787874353.77 total 60.05 last 75.66) total: 4500 open/close in 65.92 seconds: 68.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1348 (time 1787874373.51 total 10.01 last 134.70) - /unlink 2597 (time 1787874383.51 total 20.01 last 124.88) - /unlink 3872 (time 1787874393.52 total 30.01 last 127.43) total: 4500 /unlink in 34.87 seconds: 129.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 668 (time 1787874412.94 total 10.03 last 66.61) - open/close 1383 (time 1787874422.95 total 20.03 last 71.47) - open/close 2058 (time 1787874432.95 total 30.03 last 67.50) - open/close 2741 (time 1787874442.95 total 40.03 last 68.30) - open/close 3492 (time 1787874452.98 total 50.07 last 74.86) - open/close 4132 (time 1787874462.99 total 60.07 last 63.94) total: 4500 open/close in 65.67 seconds: 68.52 ops/second - /unlink 1262 (time 1787874482.51 total 10.01 last 126.13) - /unlink 2453 (time 1787874492.51 total 20.01 last 119.03) - /unlink 3812 (time 1787874502.51 total 30.01 last 135.90) total: 4500 /unlink in 34.93 seconds: 128.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 744 (time 1787874521.56 total 10.03 last 74.18) - open/close 1395 (time 1787874531.57 total 20.04 last 65.04) - open/close 2106 (time 1787874541.59 total 30.06 last 70.98) - open/close 2785 (time 1787874551.62 total 40.08 last 67.71) - open/close 3422 (time 1787874561.63 total 50.10 last 63.63) - open/close 4100 (time 1787874571.63 total 60.10 last 67.79) total: 4500 open/close in 65.35 seconds: 68.86 ops/second - /unlink 1154 (time 1787874592.19 total 10.02 last 115.22) - /unlink 2173 (time 1787874602.20 total 20.03 last 101.75) - /unlink 3566 (time 1787874612.21 total 30.04 last 139.17) total: 4500 /unlink in 38.24 seconds: 117.69 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11375: 95195 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg360-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg360-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg360-server unloading modules via unload_modules_local on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing unload_modules_local oleg360-server: modules unloaded.