== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:34:35 (1787783675) Stopping clients: oleg435-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg435-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg435-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg435-server oleg435-server: oleg435-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg435-server: detected 4 online CPUs by sysfs oleg435-server: MODOPTS_LIBCFS= oleg435-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg435-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg435-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg435-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 684 (time 1787783898.77 total 10.01 last 68.31) - open/close 1306 (time 1787783908.77 total 20.02 last 62.18) - open/close 1877 (time 1787783918.81 total 30.06 last 56.86) - open/close 2431 (time 1787783928.84 total 40.08 last 55.26) - open/close 3053 (time 1787783938.85 total 50.09 last 62.14) - open/close 3680 (time 1787783948.86 total 60.11 last 62.62) - open/close 4295 (time 1787783958.87 total 70.12 last 61.43) total: 4500 open/close in 73.67 seconds: 61.09 ops/second - /unlink 1215 (time 1787783977.58 total 10.00 last 121.49) - /unlink 2522 (time 1787783987.58 total 20.00 last 130.69) - /unlink 3763 (time 1787783997.59 total 30.00 last 124.08) total: 4500 /unlink in 35.87 seconds: 125.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 677 (time 1787784018.60 total 10.01 last 67.63) - open/close 1283 (time 1787784028.62 total 20.02 last 60.51) - open/close 1892 (time 1787784038.62 total 30.03 last 60.86) - open/close 2501 (time 1787784048.63 total 40.04 last 60.87) - open/close 3124 (time 1787784058.64 total 50.04 last 62.25) - open/close 3834 (time 1787784068.64 total 60.05 last 70.96) total: 4500 open/close in 69.37 seconds: 64.87 ops/second - /unlink 1113 (time 1787784092.68 total 10.01 last 111.17) - /unlink 2212 (time 1787784102.68 total 20.01 last 109.87) - /unlink 3347 (time 1787784112.68 total 30.02 last 113.47) - /unlink 4457 (time 1787784122.69 total 40.02 last 110.93) total: 4500 /unlink in 40.65 seconds: 110.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 597 (time 1787784138.96 total 10.00 last 59.67) - open/close 1146 (time 1787784148.96 total 20.01 last 54.88) - open/close 1707 (time 1787784158.97 total 30.02 last 56.05) - open/close 2201 (time 1787784168.98 total 40.02 last 49.38) - open/close 2780 (time 1787784179.00 total 50.05 last 57.76) - open/close 3352 (time 1787784189.00 total 60.05 last 57.17) - open/close 3993 (time 1787784199.02 total 70.06 last 64.02) total: 4500 open/close in 78.43 seconds: 57.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1222 (time 1787784222.26 total 10.00 last 122.20) - /unlink 2411 (time 1787784232.26 total 20.00 last 118.86) - /unlink 3597 (time 1787784242.29 total 30.03 last 118.29) total: 4500 /unlink in 37.41 seconds: 120.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 644 (time 1787784266.32 total 10.00 last 64.39) - open/close 1291 (time 1787784276.33 total 20.01 last 64.62) - open/close 1886 (time 1787784286.34 total 30.03 last 59.42) - open/close 2555 (time 1787784296.35 total 40.03 last 66.87) - open/close 3180 (time 1787784306.36 total 50.04 last 62.45) - open/close 3713 (time 1787784316.37 total 60.05 last 53.24) - open/close 4302 (time 1787784326.38 total 70.06 last 58.85) total: 4500 open/close in 72.77 seconds: 61.84 ops/second - /unlink 1239 (time 1787784344.08 total 10.00 last 123.85) - /unlink 2358 (time 1787784354.08 total 20.01 last 111.84) - /unlink 3508 (time 1787784364.08 total 30.01 last 114.98) total: 4500 /unlink in 38.73 seconds: 116.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 533 (time 1787784389.01 total 10.01 last 53.25) - open/close 1098 (time 1787784399.04 total 20.04 last 56.31) - open/close 1665 (time 1787784409.06 total 30.07 last 56.57) - open/close 2375 (time 1787784419.07 total 40.08 last 70.93) - open/close 2963 (time 1787784429.08 total 50.08 last 58.75) - open/close 3548 (time 1787784439.09 total 60.09 last 58.47) - open/close 4186 (time 1787784449.10 total 70.11 last 63.70) total: 4500 open/close in 74.88 seconds: 60.10 ops/second - /unlink 1262 (time 1787784468.63 total 10.01 last 126.13) - /unlink 2463 (time 1787784478.63 total 20.01 last 120.08) - /unlink 3684 (time 1787784488.65 total 30.03 last 121.84) total: 4500 /unlink in 36.78 seconds: 122.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 544 (time 1787784510.91 total 10.01 last 54.36) - open/close 1229 (time 1787784520.92 total 20.02 last 68.41) - open/close 1890 (time 1787784530.93 total 30.03 last 66.01) - open/close 2604 (time 1787784540.94 total 40.04 last 71.33) - open/close 3284 (time 1787784550.95 total 50.05 last 67.94) - open/close 4013 (time 1787784560.96 total 60.06 last 72.85) total: 4500 open/close in 68.02 seconds: 66.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1176 (time 1787784584.82 total 10.00 last 117.58) - /unlink 2315 (time 1787784594.82 total 20.01 last 113.84) - /unlink 3556 (time 1787784604.82 total 30.01 last 124.05) total: 4500 /unlink in 38.42 seconds: 117.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 549 (time 1787784629.34 total 10.02 last 54.76) - open/close 1141 (time 1787784639.34 total 20.03 last 59.17) - open/close 1630 (time 1787784649.37 total 30.05 last 48.78) - open/close 2160 (time 1787784659.37 total 40.06 last 52.98) - open/close 2713 (time 1787784669.38 total 50.07 last 55.24) - open/close 3267 (time 1787784679.39 total 60.08 last 55.33) - open/close 3865 (time 1787784689.40 total 70.09 last 59.77) total: 4500 open/close in 79.65 seconds: 56.50 ops/second - /unlink 1207 (time 1787784714.03 total 10.00 last 120.65) - /unlink 2459 (time 1787784724.04 total 20.02 last 125.06) - /unlink 3713 (time 1787784734.05 total 30.03 last 125.19) total: 4500 /unlink in 36.18 seconds: 124.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 619 (time 1787784755.89 total 10.01 last 61.82) - open/close 1263 (time 1787784765.90 total 20.02 last 64.35) - open/close 1887 (time 1787784775.90 total 30.03 last 62.37) - open/close 2487 (time 1787784785.91 total 40.03 last 59.95) - open/close 3163 (time 1787784795.93 total 50.05 last 67.47) - open/close 3783 (time 1787784805.93 total 60.05 last 61.99) total: 4500 open/close in 69.95 seconds: 64.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1285 (time 1787784830.79 total 10.00 last 128.48) - /unlink 2523 (time 1787784840.80 total 20.01 last 123.66) - /unlink 3801 (time 1787784850.81 total 30.02 last 127.72) total: 4500 /unlink in 36.54 seconds: 123.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 572 (time 1787784872.95 total 10.00 last 57.18) - open/close 1124 (time 1787784882.96 total 20.01 last 55.16) - open/close 1759 (time 1787784892.97 total 30.02 last 63.46) - open/close 2328 (time 1787784902.97 total 40.02 last 56.87) - open/close 2928 (time 1787784912.98 total 50.03 last 59.93) - open/close 3556 (time 1787784922.99 total 60.04 last 62.74) - open/close 4248 (time 1787784933.00 total 70.05 last 69.13) total: 4500 open/close in 73.48 seconds: 61.24 ops/second - /unlink 1210 (time 1787784951.44 total 10.00 last 120.96) - /unlink 2462 (time 1787784961.46 total 20.02 last 124.97) - /unlink 3725 (time 1787784971.46 total 30.02 last 126.27) total: 4500 /unlink in 36.41 seconds: 123.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 678 (time 1787784992.75 total 10.03 last 67.62) - open/close 1355 (time 1787785002.75 total 20.03 last 67.69) - open/close 2075 (time 1787785012.76 total 30.04 last 71.90) - open/close 2757 (time 1787785022.77 total 40.05 last 68.15) - open/close 3470 (time 1787785032.78 total 50.06 last 71.26) - open/close 4165 (time 1787785042.79 total 60.07 last 69.39) total: 4500 open/close in 65.21 seconds: 69.01 ops/second - /unlink 1151 (time 1787785061.96 total 10.01 last 114.93) - /unlink 2191 (time 1787785071.97 total 20.02 last 103.98) - /unlink 3298 (time 1787785081.97 total 30.02 last 110.69) - /unlink 4312 (time 1787785091.99 total 40.04 last 101.13) total: 4500 /unlink in 42.15 seconds: 106.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 505 (time 1787785110.14 total 10.02 last 50.42) - open/close 1062 (time 1787785120.16 total 20.03 last 55.62) - open/close 1675 (time 1787785130.16 total 30.03 last 61.30) - open/close 2221 (time 1787785140.18 total 40.05 last 54.51) - open/close 2775 (time 1787785150.18 total 50.05 last 55.40) - open/close 3431 (time 1787785160.18 total 60.05 last 65.56) - open/close 4109 (time 1787785170.19 total 70.06 last 67.74) total: 4500 open/close in 75.95 seconds: 59.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1059 (time 1787785191.64 total 10.00 last 105.86) - /unlink 2162 (time 1787785201.64 total 20.01 last 110.25) - /unlink 3487 (time 1787785211.65 total 30.01 last 132.45) total: 4500 /unlink in 38.77 seconds: 116.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 720 (time 1787785235.79 total 10.01 last 71.93) - open/close 1315 (time 1787785245.80 total 20.02 last 59.45) - open/close 1858 (time 1787785255.80 total 30.02 last 54.28) - open/close 2425 (time 1787785265.81 total 40.03 last 56.67) - open/close 2971 (time 1787785275.81 total 50.03 last 54.57) - open/close 3529 (time 1787785285.82 total 60.04 last 55.74) - open/close 4093 (time 1787785295.84 total 70.05 last 56.34) total: 4500 open/close in 75.52 seconds: 59.59 ops/second - /unlink 1378 (time 1787785316.17 total 10.00 last 137.78) - /unlink 2605 (time 1787785326.18 total 20.01 last 122.57) - /unlink 3728 (time 1787785336.19 total 30.02 last 112.26) total: 4500 /unlink in 36.74 seconds: 122.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 519 (time 1787785358.98 total 10.01 last 51.87) - open/close 1080 (time 1787785368.99 total 20.01 last 56.05) - open/close 1566 (time 1787785378.99 total 30.02 last 48.57) - open/close 2080 (time 1787785389.01 total 40.04 last 51.32) - open/close 2534 (time 1787785399.01 total 50.04 last 45.39) - open/close 3226 (time 1787785409.02 total 60.05 last 69.12) - open/close 3856 (time 1787785419.03 total 70.06 last 62.95) - open/close 4493 (time 1787785429.03 total 80.06 last 63.69) total: 4500 open/close in 80.18 seconds: 56.13 ops/second - /unlink 1129 (time 1787785444.88 total 10.00 last 112.87) - /unlink 2234 (time 1787785454.88 total 20.01 last 110.46) - /unlink 3481 (time 1787785464.89 total 30.01 last 124.61) total: 4500 /unlink in 39.46 seconds: 114.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 628 (time 1787785490.02 total 10.01 last 62.74) - open/close 1288 (time 1787785500.03 total 20.02 last 65.91) - open/close 1897 (time 1787785510.04 total 30.03 last 60.87) - open/close 2562 (time 1787785520.05 total 40.04 last 66.42) - open/close 3171 (time 1787785530.06 total 50.05 last 60.83) - open/close 3783 (time 1787785540.07 total 60.06 last 61.15) - open/close 4393 (time 1787785550.08 total 70.07 last 60.92) total: 4500 open/close in 71.73 seconds: 62.73 ops/second - /unlink 1240 (time 1787785567.00 total 10.00 last 123.96) - /unlink 2414 (time 1787785577.00 total 20.00 last 117.39) - /unlink 3791 (time 1787785587.00 total 30.00 last 137.69) total: 4500 /unlink in 36.19 seconds: 124.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 648 (time 1787785607.85 total 10.01 last 64.74) - open/close 1370 (time 1787785617.85 total 20.01 last 72.18) - open/close 2035 (time 1787785627.85 total 30.01 last 66.50) - open/close 2650 (time 1787785637.86 total 40.02 last 61.46) - open/close 3263 (time 1787785647.88 total 50.04 last 61.15) - open/close 3904 (time 1787785657.89 total 60.05 last 64.05) total: 4500 open/close in 68.58 seconds: 65.62 ops/second - /unlink 1199 (time 1787785682.45 total 10.00 last 119.86) - /unlink 2349 (time 1787785692.46 total 20.01 last 114.87) - /unlink 3456 (time 1787785702.47 total 30.02 last 110.62) total: 4500 /unlink in 39.81 seconds: 113.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 566 (time 1787785728.88 total 10.02 last 56.51) - open/close 1177 (time 1787785738.90 total 20.03 last 61.01) - open/close 1831 (time 1787785748.90 total 30.03 last 65.38) - open/close 2440 (time 1787785758.93 total 40.06 last 60.74) - open/close 3101 (time 1787785768.94 total 50.07 last 66.02) - open/close 3755 (time 1787785778.94 total 60.07 last 65.39) - open/close 4396 (time 1787785788.96 total 70.09 last 64.01) total: 4500 open/close in 71.96 seconds: 62.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1045 (time 1787785805.71 total 10.00 last 104.49) - /unlink 2120 (time 1787785815.72 total 20.01 last 107.37) - /unlink 3273 (time 1787785825.72 total 30.01 last 115.28) - /unlink 4490 (time 1787785835.73 total 40.02 last 121.60) total: 4500 /unlink in 40.10 seconds: 112.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 718 (time 1787785850.53 total 10.01 last 71.72) - open/close 1399 (time 1787785860.53 total 20.01 last 68.09) - open/close 2074 (time 1787785870.57 total 30.05 last 67.27) - open/close 2685 (time 1787785880.59 total 40.07 last 60.94) - open/close 3288 (time 1787785890.60 total 50.08 last 60.25) - open/close 3948 (time 1787785900.61 total 60.09 last 65.93) total: 4500 open/close in 67.96 seconds: 66.21 ops/second - /unlink 1226 (time 1787785922.90 total 10.00 last 122.57) - /unlink 2462 (time 1787785932.90 total 20.00 last 123.59) - /unlink 3728 (time 1787785942.90 total 30.01 last 126.57) total: 4500 /unlink in 36.07 seconds: 124.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 672 (time 1787785963.63 total 10.01 last 67.15) - open/close 1307 (time 1787785973.64 total 20.01 last 63.48) - open/close 1985 (time 1787785983.66 total 30.03 last 67.64) - open/close 2676 (time 1787785993.66 total 40.03 last 69.09) - open/close 3333 (time 1787786003.67 total 50.04 last 65.68) - open/close 3929 (time 1787786013.68 total 60.05 last 59.52) total: 4500 open/close in 67.84 seconds: 66.33 ops/second - /unlink 1285 (time 1787786036.02 total 10.00 last 128.48) - /unlink 2669 (time 1787786046.03 total 20.01 last 138.27) - /unlink 4037 (time 1787786056.04 total 30.02 last 136.69) total: 4500 /unlink in 33.12 seconds: 135.87 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95361 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg435-server unloading modules via unload_modules_local on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing unload_modules_local oleg435-server: modules unloaded.