== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:55:28 (1783950928) Stopping clients: oleg309-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg309-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg309-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg309-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg309-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg309-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg309-server oleg309-server: oleg309-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg309-server' oleg309-server: oleg309-server.virtnet: executing load_modules_local oleg309-server: Loading modules from /home/green/git/lustre-release/lustre oleg309-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg309-server: detected 4 online CPUs by sysfs oleg309-server: MODOPTS_LIBCFS= oleg309-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg309-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg309-server: oleg309-server.virtnet: executing set_default_debug -1 all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg309-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg309-server: oleg309-server.virtnet: executing set_default_debug -1 all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg309-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug -1 all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 713 (time 1783951131.51 total 10.01 last 71.21) - open/close 1497 (time 1783951141.51 total 20.02 last 78.37) - open/close 2194 (time 1783951151.51 total 30.02 last 69.66) - open/close 2933 (time 1783951161.52 total 40.03 last 73.83) - open/close 3601 (time 1783951171.53 total 50.04 last 66.74) - open/close 4218 (time 1783951181.54 total 60.05 last 61.63) total: 4500 open/close in 64.59 seconds: 69.67 ops/second - /unlink 1223 (time 1783951201.00 total 10.01 last 122.21) - /unlink 2513 (time 1783951211.00 total 20.02 last 128.90) - /unlink 3798 (time 1783951221.01 total 30.02 last 128.44) total: 4500 /unlink in 36.38 seconds: 123.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 516 (time 1783951242.44 total 10.00 last 51.59) - open/close 1054 (time 1783951252.44 total 20.01 last 53.76) - open/close 1744 (time 1783951262.45 total 30.01 last 68.99) - open/close 2408 (time 1783951272.45 total 40.02 last 66.34) - open/close 3085 (time 1783951282.47 total 50.03 last 67.61) - open/close 3762 (time 1783951292.47 total 60.04 last 67.69) - open/close 4403 (time 1783951302.47 total 70.04 last 64.09) total: 4500 open/close in 71.62 seconds: 62.83 ops/second - /unlink 1136 (time 1783951319.03 total 10.02 last 113.38) - /unlink 2320 (time 1783951329.04 total 20.02 last 118.34) - /unlink 3535 (time 1783951339.04 total 30.03 last 121.46) total: 4500 /unlink in 38.80 seconds: 115.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 663 (time 1783951362.37 total 10.01 last 66.21) - open/close 1257 (time 1783951372.38 total 20.02 last 59.37) - open/close 1929 (time 1783951382.38 total 30.02 last 67.17) - open/close 2597 (time 1783951392.39 total 40.03 last 66.76) - open/close 3286 (time 1783951402.39 total 50.03 last 68.89) - open/close 3985 (time 1783951412.40 total 60.05 last 69.81) total: 4500 open/close in 68.49 seconds: 65.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1222 (time 1783951435.28 total 10.00 last 122.15) - /unlink 2373 (time 1783951445.29 total 20.01 last 115.05) - /unlink 3515 (time 1783951455.30 total 30.02 last 114.05) total: 4500 /unlink in 38.27 seconds: 117.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 600 (time 1783951478.68 total 10.01 last 59.92) - open/close 1304 (time 1783951488.69 total 20.02 last 70.36) - open/close 1966 (time 1783951498.70 total 30.03 last 66.13) - open/close 2653 (time 1783951508.71 total 40.04 last 68.63) - open/close 3310 (time 1783951518.72 total 50.05 last 65.65) - open/close 3977 (time 1783951528.73 total 60.06 last 66.64) total: 4500 open/close in 67.46 seconds: 66.70 ops/second - /unlink 1208 (time 1783951550.31 total 10.01 last 120.72) - /unlink 2452 (time 1783951560.31 total 20.01 last 124.34) - /unlink 3737 (time 1783951570.32 total 30.02 last 128.45) total: 4500 /unlink in 36.59 seconds: 122.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 617 (time 1783951591.86 total 10.00 last 61.67) - open/close 1277 (time 1783951601.87 total 20.02 last 65.92) - open/close 1937 (time 1783951611.89 total 30.03 last 65.91) - open/close 2614 (time 1783951621.89 total 40.03 last 67.69) - open/close 3240 (time 1783951631.91 total 50.05 last 62.49) - open/close 3875 (time 1783951641.92 total 60.06 last 63.42) total: 4500 open/close in 69.34 seconds: 64.90 ops/second - /unlink 1304 (time 1783951665.45 total 10.01 last 130.29) - /unlink 2382 (time 1783951675.45 total 20.01 last 107.79) - /unlink 3540 (time 1783951685.46 total 30.02 last 115.71) total: 4500 /unlink in 38.36 seconds: 117.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 606 (time 1783951709.55 total 10.00 last 60.59) - open/close 1302 (time 1783951719.55 total 20.01 last 69.58) - open/close 2010 (time 1783951729.55 total 30.01 last 70.80) - open/close 2653 (time 1783951739.55 total 40.01 last 64.27) - open/close 3357 (time 1783951749.56 total 50.02 last 70.33) - open/close 4032 (time 1783951759.59 total 60.05 last 67.33) total: 4500 open/close in 66.81 seconds: 67.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1261 (time 1783951781.12 total 10.01 last 125.98) - /unlink 2391 (time 1783951791.12 total 20.01 last 112.96) - /unlink 3597 (time 1783951801.13 total 30.02 last 120.54) total: 4500 /unlink in 37.17 seconds: 121.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 462 (time 1783951823.94 total 10.01 last 46.14) - open/close 1001 (time 1783951833.95 total 20.02 last 53.85) - open/close 1605 (time 1783951843.96 total 30.03 last 60.35) - open/close 2246 (time 1783951853.96 total 40.03 last 64.09) - open/close 2963 (time 1783951863.99 total 50.05 last 71.53) - open/close 3675 (time 1783951874.00 total 60.07 last 71.11) - open/close 4379 (time 1783951884.01 total 70.07 last 70.36) total: 4500 open/close in 71.77 seconds: 62.70 ops/second - /unlink 1303 (time 1783951900.10 total 10.01 last 130.17) - /unlink 2498 (time 1783951910.10 total 20.01 last 119.49) - /unlink 3627 (time 1783951920.11 total 30.01 last 112.85) total: 4500 /unlink in 36.98 seconds: 121.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 713 (time 1783951942.60 total 10.00 last 71.28) - open/close 1441 (time 1783951952.61 total 20.01 last 72.76) - open/close 2133 (time 1783951962.62 total 30.02 last 69.11) - open/close 2803 (time 1783951972.63 total 40.02 last 66.98) - open/close 3425 (time 1783951982.63 total 50.03 last 62.19) - open/close 4069 (time 1783951992.64 total 60.04 last 64.31) total: 4500 open/close in 66.56 seconds: 67.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1257 (time 1783952013.93 total 10.01 last 125.62) - /unlink 2414 (time 1783952023.94 total 20.01 last 115.66) - /unlink 3587 (time 1783952033.94 total 30.01 last 117.29) total: 4500 /unlink in 37.74 seconds: 119.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 548 (time 1783952056.45 total 10.01 last 54.74) - open/close 1154 (time 1783952066.47 total 20.03 last 60.48) - open/close 1770 (time 1783952076.47 total 30.04 last 61.56) - open/close 2431 (time 1783952086.47 total 40.04 last 66.08) - open/close 3061 (time 1783952096.48 total 50.05 last 62.94) - open/close 3785 (time 1783952106.51 total 60.08 last 72.20) - open/close 4420 (time 1783952116.53 total 70.09 last 63.40) total: 4500 open/close in 71.40 seconds: 63.02 ops/second - /unlink 1245 (time 1783952132.18 total 10.00 last 124.44) - /unlink 2480 (time 1783952142.19 total 20.01 last 123.44) - /unlink 3650 (time 1783952152.19 total 30.01 last 116.95) total: 4500 /unlink in 37.95 seconds: 118.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 527 (time 1783952176.05 total 10.01 last 52.66) - open/close 1041 (time 1783952186.06 total 20.02 last 51.35) - open/close 1646 (time 1783952196.07 total 30.03 last 60.40) - open/close 2241 (time 1783952206.10 total 40.06 last 59.32) - open/close 2807 (time 1783952216.12 total 50.08 last 56.53) - open/close 3492 (time 1783952226.13 total 60.08 last 68.43) - open/close 4126 (time 1783952236.13 total 70.09 last 63.40) total: 4500 open/close in 75.79 seconds: 59.37 ops/second - /unlink 1117 (time 1783952257.38 total 10.01 last 111.61) - /unlink 2236 (time 1783952267.38 total 20.01 last 111.85) - /unlink 3350 (time 1783952277.41 total 30.03 last 111.15) - /unlink 4478 (time 1783952287.41 total 40.04 last 112.79) total: 4500 /unlink in 40.28 seconds: 111.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 543 (time 1783952302.76 total 10.01 last 54.25) - open/close 1034 (time 1783952312.79 total 20.04 last 48.96) - open/close 1617 (time 1783952322.80 total 30.04 last 58.27) - open/close 2233 (time 1783952332.82 total 40.06 last 61.47) - open/close 2832 (time 1783952342.83 total 50.08 last 59.83) - open/close 3507 (time 1783952352.83 total 60.08 last 67.47) - open/close 4128 (time 1783952362.85 total 70.09 last 62.02) total: 4500 open/close in 76.38 seconds: 58.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1210 (time 1783952384.06 total 10.00 last 120.98) - /unlink 2464 (time 1783952394.06 total 20.00 last 125.39) - /unlink 3706 (time 1783952404.07 total 30.01 last 124.14) total: 4500 /unlink in 36.85 seconds: 122.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 709 (time 1783952426.00 total 10.05 last 70.57) - open/close 1318 (time 1783952436.01 total 20.05 last 60.86) - open/close 1911 (time 1783952446.01 total 30.06 last 59.25) - open/close 2443 (time 1783952456.02 total 40.07 last 53.18) - open/close 3107 (time 1783952466.02 total 50.07 last 66.39) - open/close 3767 (time 1783952476.02 total 60.07 last 65.99) - open/close 4422 (time 1783952486.02 total 70.07 last 65.48) total: 4500 open/close in 71.21 seconds: 63.19 ops/second - /unlink 1195 (time 1783952502.26 total 10.00 last 119.47) - /unlink 2368 (time 1783952512.26 total 20.01 last 117.24) - /unlink 3587 (time 1783952522.27 total 30.02 last 121.73) total: 4500 /unlink in 38.20 seconds: 117.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 676 (time 1783952546.14 total 10.01 last 67.55) - open/close 1367 (time 1783952556.16 total 20.03 last 68.96) - open/close 2057 (time 1783952566.18 total 30.04 last 68.88) - open/close 2755 (time 1783952576.19 total 40.05 last 69.73) - open/close 3444 (time 1783952586.20 total 50.06 last 68.84) - open/close 4068 (time 1783952596.20 total 60.06 last 62.39) total: 4500 open/close in 67.57 seconds: 66.60 ops/second - /unlink 1144 (time 1783952619.25 total 10.01 last 114.29) - /unlink 2148 (time 1783952629.26 total 20.02 last 100.29) - /unlink 3392 (time 1783952639.27 total 30.03 last 124.32) total: 4500 /unlink in 39.40 seconds: 114.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 608 (time 1783952664.83 total 10.03 last 60.63) - open/close 1246 (time 1783952674.85 total 20.05 last 63.65) - open/close 1961 (time 1783952684.86 total 30.05 last 71.47) - open/close 2588 (time 1783952694.86 total 40.06 last 62.69) - open/close 3324 (time 1783952704.86 total 50.06 last 73.59) - open/close 3972 (time 1783952714.87 total 60.07 last 64.71) total: 4500 open/close in 68.55 seconds: 65.64 ops/second - /unlink 1380 (time 1783952738.60 total 10.00 last 137.95) - /unlink 2570 (time 1783952748.62 total 20.02 last 118.82) - /unlink 3677 (time 1783952758.62 total 30.02 last 110.64) total: 4500 /unlink in 37.30 seconds: 120.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 741 (time 1783952781.71 total 10.00 last 74.07) - open/close 1407 (time 1783952791.73 total 20.03 last 66.41) - open/close 2068 (time 1783952801.74 total 30.04 last 66.03) - open/close 2762 (time 1783952811.76 total 40.06 last 69.31) - open/close 3425 (time 1783952821.76 total 50.06 last 66.27) - open/close 4146 (time 1783952831.78 total 60.08 last 71.99) total: 4500 open/close in 65.25 seconds: 68.97 ops/second - /unlink 1062 (time 1783952851.63 total 10.00 last 106.17) - /unlink 2195 (time 1783952861.63 total 20.01 last 113.25) - /unlink 3365 (time 1783952871.65 total 30.02 last 116.80) total: 4500 /unlink in 38.84 seconds: 115.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 734 (time 1783952895.89 total 10.01 last 73.35) - open/close 1441 (time 1783952905.90 total 20.02 last 70.63) - open/close 2092 (time 1783952915.90 total 30.02 last 65.08) - open/close 2802 (time 1783952925.92 total 40.04 last 70.89) - open/close 3520 (time 1783952935.94 total 50.06 last 71.66) - open/close 4090 (time 1783952945.94 total 60.06 last 56.99) total: 4500 open/close in 67.63 seconds: 66.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 993 (time 1783952969.79 total 10.02 last 99.10) - /unlink 2024 (time 1783952979.80 total 20.04 last 102.92) - /unlink 3132 (time 1783952989.81 total 30.04 last 110.76) - /unlink 4318 (time 1783952999.81 total 40.04 last 118.59) total: 4500 /unlink in 42.01 seconds: 107.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 662 (time 1783953016.85 total 10.00 last 66.18) - open/close 1304 (time 1783953026.85 total 20.01 last 64.18) - open/close 1992 (time 1783953036.86 total 30.01 last 68.77) - open/close 2675 (time 1783953046.88 total 40.03 last 68.16) - open/close 3368 (time 1783953056.88 total 50.04 last 69.26) - open/close 4024 (time 1783953066.89 total 60.04 last 65.55) total: 4500 open/close in 69.54 seconds: 64.71 ops/second - /unlink 1078 (time 1783953091.25 total 10.00 last 107.78) - /unlink 2216 (time 1783953101.27 total 20.02 last 113.58) - /unlink 3291 (time 1783953111.27 total 30.02 last 107.45) - /unlink 4385 (time 1783953121.28 total 40.03 last 109.35) total: 4500 /unlink in 41.55 seconds: 108.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 560 (time 1783953138.30 total 10.00 last 55.98) - open/close 1233 (time 1783953148.31 total 20.01 last 67.22) - open/close 1873 (time 1783953158.32 total 30.02 last 63.95) - open/close 2545 (time 1783953168.32 total 40.03 last 67.18) - open/close 3195 (time 1783953178.33 total 50.03 last 64.95) - open/close 3861 (time 1783953188.34 total 60.05 last 66.50) - open/close 4489 (time 1783953198.37 total 70.07 last 62.65) total: 4500 open/close in 70.33 seconds: 63.98 ops/second - /unlink 1214 (time 1783953213.42 total 10.00 last 121.37) - /unlink 2422 (time 1783953223.43 total 20.01 last 120.70) - /unlink 3613 (time 1783953233.45 total 30.03 last 118.82) total: 4500 /unlink in 37.47 seconds: 120.08 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 95352 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg309-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg309-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg309-server stop mds service on oleg309-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg309-server stop mds service on oleg309-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg309-server unloading modules via unload_modules_local on: 'oleg309-server' oleg309-server: oleg309-server.virtnet: executing unload_modules_local oleg309-server: modules unloaded.