== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:44:08 (1763340248) Stopping clients: oleg116-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg116-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg116-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg116-server oleg116-server: oleg116-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg116-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg116-server: detected 4 online CPUs by sysfs oleg116-server: MODOPTS_LIBCFS= oleg116-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg116-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg116-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg116-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 811 (time 1763340398.81 total 10.00 last 81.08) - open/close 1580 (time 1763340408.81 total 20.00 last 76.90) - open/close 2368 (time 1763340418.82 total 30.01 last 78.77) - open/close 3233 (time 1763340428.82 total 40.01 last 86.47) - open/close 4085 (time 1763340438.83 total 50.02 last 85.11) total: 4500 open/close in 55.12 seconds: 81.64 ops/second - /unlink 1381 (time 1763340457.53 total 10.01 last 138.03) - /unlink 2883 (time 1763340467.54 total 20.01 last 150.12) - /unlink 4365 (time 1763340477.55 total 30.02 last 148.10) total: 4500 /unlink in 31.12 seconds: 144.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 924 (time 1763340491.92 total 10.01 last 92.29) - open/close 1860 (time 1763340501.92 total 20.02 last 93.57) - open/close 2746 (time 1763340511.94 total 30.03 last 88.45) - open/close 3623 (time 1763340521.95 total 40.04 last 87.59) - open/close 4391 (time 1763340531.95 total 50.05 last 76.77) total: 4500 open/close in 51.30 seconds: 87.71 ops/second - /unlink 1317 (time 1763340546.13 total 10.01 last 131.55) - /unlink 2887 (time 1763340556.14 total 20.02 last 156.91) - /unlink 4405 (time 1763340566.14 total 30.02 last 151.74) total: 4500 /unlink in 30.64 seconds: 146.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 835 (time 1763340580.30 total 10.00 last 83.47) - open/close 1664 (time 1763340590.30 total 20.01 last 82.87) - open/close 2528 (time 1763340600.31 total 30.01 last 86.33) - open/close 3368 (time 1763340610.33 total 40.03 last 83.87) - open/close 4211 (time 1763340620.33 total 50.03 last 84.27) total: 4500 open/close in 53.56 seconds: 84.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1501 (time 1763340637.15 total 10.00 last 150.06) - /unlink 2863 (time 1763340647.15 total 20.00 last 136.17) - /unlink 4246 (time 1763340657.15 total 30.01 last 138.23) total: 4500 /unlink in 31.76 seconds: 141.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 791 (time 1763340672.03 total 10.01 last 79.03) - open/close 1574 (time 1763340682.04 total 20.03 last 78.17) - open/close 2358 (time 1763340692.05 total 30.03 last 78.38) - open/close 3146 (time 1763340702.06 total 40.04 last 78.69) - open/close 3958 (time 1763340712.07 total 50.05 last 81.11) total: 4500 open/close in 57.61 seconds: 78.12 ops/second - /unlink 1514 (time 1763340732.69 total 10.01 last 151.32) - /unlink 2853 (time 1763340742.69 total 20.01 last 133.87) - /unlink 4102 (time 1763340752.70 total 30.02 last 124.79) total: 4500 /unlink in 33.12 seconds: 135.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 754 (time 1763340769.02 total 10.00 last 75.39) - open/close 1511 (time 1763340779.03 total 20.00 last 75.68) - open/close 2293 (time 1763340789.03 total 30.01 last 78.19) - open/close 3042 (time 1763340799.04 total 40.01 last 74.84) - open/close 3835 (time 1763340809.05 total 50.02 last 79.20) total: 4500 open/close in 58.80 seconds: 76.52 ops/second - /unlink 1434 (time 1763340830.96 total 10.00 last 143.35) - /unlink 2780 (time 1763340840.97 total 20.01 last 134.51) - /unlink 4134 (time 1763340850.97 total 30.02 last 135.31) total: 4500 /unlink in 32.96 seconds: 136.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 687 (time 1763340868.41 total 10.00 last 68.68) - open/close 1403 (time 1763340878.41 total 20.00 last 71.60) - open/close 2209 (time 1763340888.41 total 30.01 last 80.58) - open/close 3004 (time 1763340898.41 total 40.01 last 79.50) - open/close 3769 (time 1763340908.42 total 50.01 last 76.46) total: 4500 open/close in 59.48 seconds: 75.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1453 (time 1763340931.63 total 10.01 last 145.17) - /unlink 2774 (time 1763340941.64 total 20.02 last 132.02) - /unlink 4207 (time 1763340951.65 total 30.02 last 143.23) total: 4500 /unlink in 32.53 seconds: 138.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 812 (time 1763340967.74 total 10.00 last 81.18) - open/close 1597 (time 1763340977.75 total 20.01 last 78.47) - open/close 2244 (time 1763340987.77 total 30.03 last 64.52) - open/close 2895 (time 1763340997.78 total 40.04 last 65.08) - open/close 3623 (time 1763341007.78 total 50.04 last 72.77) - open/close 4368 (time 1763341017.79 total 60.05 last 74.40) total: 4500 open/close in 61.69 seconds: 72.94 ops/second - /unlink 1293 (time 1763341033.14 total 10.00 last 129.26) - /unlink 2552 (time 1763341043.15 total 20.01 last 125.82) - /unlink 3892 (time 1763341053.15 total 30.01 last 133.95) total: 4500 /unlink in 34.45 seconds: 130.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 804 (time 1763341072.03 total 10.01 last 80.35) - open/close 1570 (time 1763341082.04 total 20.02 last 76.52) - open/close 2316 (time 1763341092.04 total 30.02 last 74.59) - open/close 3077 (time 1763341102.04 total 40.02 last 76.10) - open/close 3835 (time 1763341112.05 total 50.02 last 75.78) total: 4500 open/close in 59.14 seconds: 76.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1484 (time 1763341135.10 total 10.00 last 148.36) - /unlink 2727 (time 1763341145.11 total 20.01 last 124.20) - /unlink 4071 (time 1763341155.12 total 30.02 last 134.31) total: 4500 /unlink in 33.12 seconds: 135.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 691 (time 1763341172.59 total 10.01 last 69.03) - open/close 1410 (time 1763341182.60 total 20.02 last 71.83) - open/close 2081 (time 1763341192.61 total 30.03 last 67.04) - open/close 2801 (time 1763341202.61 total 40.03 last 71.97) - open/close 3502 (time 1763341212.63 total 50.05 last 69.98) - open/close 4254 (time 1763341222.63 total 60.05 last 75.20) total: 4500 open/close in 63.35 seconds: 71.03 ops/second - /unlink 1321 (time 1763341239.17 total 10.00 last 132.06) - /unlink 2595 (time 1763341249.17 total 20.01 last 127.36) - /unlink 3836 (time 1763341259.17 total 30.01 last 124.08) total: 4500 /unlink in 35.72 seconds: 125.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 701 (time 1763341279.15 total 10.01 last 70.05) - open/close 1437 (time 1763341289.16 total 20.01 last 73.54) - open/close 2066 (time 1763341299.17 total 30.03 last 62.83) - open/close 2752 (time 1763341309.17 total 40.03 last 68.59) - open/close 3466 (time 1763341319.18 total 50.03 last 71.39) - open/close 4142 (time 1763341329.18 total 60.03 last 67.56) total: 4500 open/close in 64.60 seconds: 69.66 ops/second - /unlink 1244 (time 1763341347.33 total 10.00 last 124.35) - /unlink 2484 (time 1763341357.34 total 20.01 last 123.88) - /unlink 3805 (time 1763341367.35 total 30.02 last 132.05) total: 4500 /unlink in 35.72 seconds: 125.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 691 (time 1763341388.67 total 10.02 last 68.98) - open/close 1335 (time 1763341398.70 total 20.05 last 64.21) - open/close 2082 (time 1763341408.71 total 30.05 last 74.65) - open/close 2822 (time 1763341418.72 total 40.06 last 73.95) - open/close 3542 (time 1763341428.73 total 50.07 last 71.93) - open/close 4229 (time 1763341438.73 total 60.07 last 68.67) total: 4500 open/close in 64.20 seconds: 70.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1122 (time 1763341458.85 total 10.00 last 112.18) - /unlink 2139 (time 1763341468.85 total 20.01 last 101.66) - /unlink 3353 (time 1763341478.86 total 30.01 last 121.39) total: 4500 /unlink in 39.12 seconds: 115.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 583 (time 1763341501.88 total 10.00 last 58.27) - open/close 1149 (time 1763341511.90 total 20.02 last 56.52) - open/close 1778 (time 1763341521.90 total 30.02 last 62.88) - open/close 2471 (time 1763341531.91 total 40.03 last 69.26) - open/close 3157 (time 1763341541.92 total 50.05 last 68.48) - open/close 3843 (time 1763341551.93 total 60.05 last 68.57) - open/close 4499 (time 1763341561.93 total 70.05 last 65.59) total: 4500 open/close in 70.05 seconds: 64.24 ops/second - /unlink 1289 (time 1763341575.92 total 10.00 last 128.86) - /unlink 2577 (time 1763341585.93 total 20.01 last 128.73) - /unlink 3764 (time 1763341595.93 total 30.01 last 118.64) total: 4500 /unlink in 36.45 seconds: 123.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 705 (time 1763341617.43 total 10.00 last 70.47) - open/close 1438 (time 1763341627.44 total 20.01 last 73.23) - open/close 2115 (time 1763341637.45 total 30.02 last 67.62) - open/close 2798 (time 1763341647.46 total 40.04 last 68.21) - open/close 3524 (time 1763341657.47 total 50.04 last 72.59) - open/close 4288 (time 1763341667.47 total 60.05 last 76.35) total: 4500 open/close in 62.99 seconds: 71.44 ops/second - /unlink 1261 (time 1763341684.22 total 10.02 last 125.90) - /unlink 2525 (time 1763341694.22 total 20.02 last 126.34) - /unlink 3704 (time 1763341704.22 total 30.02 last 117.90) total: 4500 /unlink in 36.22 seconds: 124.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 680 (time 1763341724.37 total 10.01 last 67.95) - open/close 1412 (time 1763341734.37 total 20.01 last 73.17) - open/close 2151 (time 1763341744.38 total 30.02 last 73.82) - open/close 2938 (time 1763341754.39 total 40.04 last 78.60) - open/close 3632 (time 1763341764.41 total 50.05 last 69.27) - open/close 4353 (time 1763341774.42 total 60.06 last 72.08) total: 4500 open/close in 62.03 seconds: 72.55 ops/second - /unlink 1366 (time 1763341790.80 total 10.00 last 136.57) - /unlink 2909 (time 1763341800.80 total 20.01 last 154.23) - /unlink 4470 (time 1763341810.80 total 30.01 last 156.09) total: 4500 /unlink in 30.28 seconds: 148.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 758 (time 1763341824.47 total 10.01 last 75.74) - open/close 1451 (time 1763341834.48 total 20.01 last 69.29) - open/close 2169 (time 1763341844.49 total 30.02 last 71.71) - open/close 2906 (time 1763341854.49 total 40.02 last 73.68) - open/close 3656 (time 1763341864.50 total 50.04 last 74.91) - open/close 4375 (time 1763341874.51 total 60.04 last 71.86) total: 4500 open/close in 61.68 seconds: 72.96 ops/second - /unlink 1292 (time 1763341889.71 total 10.00 last 129.19) - /unlink 2568 (time 1763341899.71 total 20.00 last 127.56) - /unlink 3808 (time 1763341909.72 total 30.01 last 123.93) total: 4500 /unlink in 35.22 seconds: 127.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 708 (time 1763341928.38 total 10.00 last 70.79) - open/close 1439 (time 1763341938.38 total 20.01 last 73.07) - open/close 2230 (time 1763341948.39 total 30.01 last 79.06) - open/close 2993 (time 1763341958.39 total 40.02 last 76.26) - open/close 3768 (time 1763341968.40 total 50.02 last 77.45) total: 4500 open/close in 59.53 seconds: 75.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1322 (time 1763341991.09 total 10.00 last 132.14) - /unlink 2677 (time 1763342001.10 total 20.01 last 135.44) - /unlink 3994 (time 1763342011.10 total 30.01 last 131.64) total: 4500 /unlink in 34.22 seconds: 131.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 696 (time 1763342029.05 total 10.00 last 69.57) - open/close 1431 (time 1763342039.06 total 20.01 last 73.44) - open/close 2165 (time 1763342049.06 total 30.02 last 73.37) - open/close 2928 (time 1763342059.07 total 40.02 last 76.25) - open/close 3665 (time 1763342069.08 total 50.03 last 73.64) - open/close 4424 (time 1763342079.09 total 60.04 last 75.82) total: 4500 open/close in 60.99 seconds: 73.78 ops/second - /unlink 1295 (time 1763342093.21 total 10.01 last 129.40) - /unlink 2929 (time 1763342103.22 total 20.01 last 163.38) - /unlink 4387 (time 1763342113.22 total 30.01 last 145.79) total: 4500 /unlink in 30.90 seconds: 145.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 770 (time 1763342127.45 total 10.00 last 76.98) - open/close 1533 (time 1763342137.45 total 20.01 last 76.27) - open/close 2291 (time 1763342147.47 total 30.02 last 75.71) - open/close 2962 (time 1763342157.47 total 40.03 last 67.06) - open/close 3671 (time 1763342167.48 total 50.03 last 70.86) - open/close 4402 (time 1763342177.50 total 60.05 last 72.96) total: 4500 open/close in 61.65 seconds: 72.99 ops/second - /unlink 1190 (time 1763342193.15 total 10.00 last 118.94) - /unlink 2417 (time 1763342203.15 total 20.01 last 122.69) - /unlink 3745 (time 1763342213.16 total 30.01 last 132.69) total: 4500 /unlink in 36.11 seconds: 124.61 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11098: 93953 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg116-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing unload_modules_local oleg116-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg116-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.