Compiling SSIS Projects with Team City - build

Has anyone done this? If so, what tools/techniques/approaches did you use?
Is it possible to do with installing the SQL Business Studio Version of Visual Studio?
Thanks in advance!

Got it folks...
1) Install MSBuild Extensions
2) Created a Build.Xml file as so...
<Project ToolsVersion="3.5" DefaultTargets="Default" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<Import Project="$(MSBuildExtensionsPath)\ExtensionPack\MSBuild.ExtensionPack.tasks"/>
<Target Name="Default">
<PropertyGroup>
<OutputRoot>../../../build-artifacts</OutputRoot>
</PropertyGroup>
<ItemGroup>
<SSISProjectFile Include="SSISProject.dtproj"/>
<SSISProject Include="#(SSISProjectFile)">
<OutputDirectory>$(OutputRoot)</OutputDirectory>
</SSISProject>
</ItemGroup>
<ItemGroup>
<Namespaces Include="Mynamespace">
<Prefix>DTS</Prefix>
<Uri>www.microsoft.com/SqlServer/Dts</Uri>
</Namespaces>
</ItemGroup>
<MSBuild.ExtensionPack.Xml.XmlFile
TaskAction="UpdateElement"
File="EnclarityDataImport.dtsx"
XPath="//DTS:Property[#DTS:Name='ConfigurationString']"
InnerText="$(MSBuildProjectDirectory)\EnclarityDataImport.dtsConfig"
Namespaces="#(Namespaces)"/>
<MSBuild.ExtensionPack.SqlServer.BuildDeploymentManifest InputProject="#(SSISProject)"/>
</Target>
The only trick was the last part of the build here. By default visual studio adds the absolute path to your config and connection string files for your dtsx package. Team City will use these in conjunction with MSBuild extensions to build the package so a local path will break the build because the paths to the build directories in Team City are automatically generated. So using the code above and the $(MSBuildProjectDirectory) you can twiddle the value of the path on your dtsx file so that it points to the path where your compilation is exectuing.

Like booyaa says SSIS projects don't need to be compiled, but what i have done is make the .dtconfigs configurable by the build/deployment process.
I do this so that i can run the packages on deployment in different environments. So the build will copy a template of the dtconfig file.
this contains tokens- $(Servername) $(ConnectionString)
And then i do the replacement on deployment and then execute by wrapping the dtexec in an command.
Not sure about 2012.

Related

How to use the Web Publishing Pipeline and Web Deploy (MSDEPLOY) to Publish a Console Application?

I would like to use web deploy to publish a Visual Studio "Console" application to a folder on the target system.
I have had some luck, and have been able to produce something similar to what I need, but not quite.
I've added the following to the console .csproj:
added the following projectName.wpp.targets file
<Import Project="$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v10.0\WebApplications\Microsoft.WebApplication.targets" />
and I've added the following projectName.wpp.targets:
<Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="4.0">
<PropertyGroup>
<DeployAsIisApp>false</DeployAsIisApp>
<IncludeSetAclProviderOnDestination>false</IncludeSetAclProviderOnDestination>
</PropertyGroup>
<ItemGroup>
<FilesForPackagingFromProject Include="$(IntermediateOutputPath)$(TargetFileName).config">
<DestinationRelativePath>bin\%(RecursiveDir)%(FileName)%(Extension)</DestinationRelativePath>
<FromTarget>projectName.wpp.targets</FromTarget>
</FilesForPackagingFromProject>
</ItemGroup>
</Project>
I then edit the .SetParameters.xml file as follows:
<parameters>
<setParameter name="IIS Web Application Name" value="c:\company\project" />
</parameters>
When I then deploy using the generated .cmd file, I get all the files deployed to C:\company\project\bin.
That's not bad, but I'd like to do better. In particular, I'd like to omit the "bin" folder and put all files in the "C:\company\project" folder, and I'd like to be able to specify the ACLs
Has anybody been able to work around these problems?
Ok, so here's the way how to omit the 'bin' folder.
First of all, I'd like to emphasize that all this msdeploy-related stuff is for web apps deployment, and 'bin' folder seems for me to be almost hardcoded deeply inside. So if you want to get rid of it - you have to do some dirty things. Which I did.
We'll have to change $(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v10.0\WebApplications\Microsoft.WebApplication.targets project a little bit, so it's better to change not it, but it's copy.
Steps:
1.Backup $(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v10.0\WebApplications\Microsoft.WebApplication.targets(alternatively, you could install MSBuild.Microsoft.VisualStudio.Web.targets package, redirect your csproj file to Microsoft.WebApplication.targets file obtained from package and work with it).
2. In the $(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v10.0\WebApplications\Microsoft.WebApplicaton.targets find the xml node which looks like <CopyPipelineFiles PipelineItems="#(FilesForPackagingFromProject)"(there are several ones of them, take the one from the line ~2570).
3. Comment the node out, replace with the custom one, so eventually it will look like:
<!--
<CopyPipelineFiles PipelineItems="#(FilesForPackagingFromProject)"
SourceDirectory="$(WebPublishPipelineProjectDirectory)"
TargetDirectory="$(WPPAllFilesInSingleFolder)"
SkipMetadataExcludeTrueItems="True"
UpdateItemSpec="True"
DeleteItemsMarkAsExcludeTrue ="True"
Condition="'#(FilesForPackagingFromProject)' != ''">
<Output TaskParameter="ResultPipelineItems" ItemName="_FilesForPackagingFromProjectTempory"/>
</CopyPipelineFiles>-->
<!-- Copying files to package folder in 'custom'(dirty) way -->
<CreateItem Include="$(OutputPath)\**\*.*">
<Output TaskParameter="Include" ItemName="YourFilesToCopy" />
</CreateItem>
<Copy SourceFiles="#(YourFilesToCopy)"
DestinationFiles="#(YourFilesToCopy->'$(WPPAllFilesInSingleFolder)\%(RecursiveDir)%(Filename)%(Extension)')" />
Then
4. Your projectName.wpp.targets don't have to have FilesForPackagingFromProject, so it will look like:
<!-- targets -->
<PropertyGroup>
<DeployAsIisApp>false</DeployAsIisApp>
<IncludeSetAclProviderOnDestination>false</IncludeSetAclProviderOnDestination>
</PropertyGroup>
<ItemGroup>
<!-- intentionally left blank -->
</ItemGroup>
</Project>
That's it. Worked for me(tm), tested. Let me be honest, I don't like this approach, but that was the only way I made it working in the needed way. It's up to you whether you'll use it in your project or not.
My opinion is not to use msdeploy here - it was not for you task.
Better to write msbuild-scripts from scratch or accept the 'bin' folder, and fight against the framework again once next customization is required.

Msbuild resolve project reference to custom project type

I've created a custom project type (cljproj instead of csproj) for a programming language called clojure. When the project compiles it outputs multiple .dll files as well as some dependent .clj files.
This is done by overriding the default CoreCompile target in the cljproj file. Which basically copies all files needing compilation to the bin directory and then executes a separate app to compile them.
<Target Name="CoreCompile">
<PropertyGroup>
<ClojureNamespaces>#(Compile -> '%(RelativeDir)%(Filename)', ' ')</ClojureNamespaces>
</PropertyGroup>
<Copy SourceFiles="#(Compile)" SkipUnchangedFiles="true" OverwriteReadOnlyFiles="true" DestinationFiles="#(Compile -> '$(OutDir)%(RelativeDir)%(Filename)%(Extension)')" />
<Exec WorkingDirectory="$(OutDir)" Command=""$(ClojureRuntimesDirectory)\$(ClojureVersion)\Clojure.Compile" $(ClojureNamespaces.Replace('\', '.'))" />
</Target>
I've added a reference from a c# project (csproj) to my clojure project (cljproj).
<ProjectReference Include="..\Clojure ASP.Net MVC Controller Library1\Clojure ASP.Net MVC Controller Library1.cljproj">
<Project>{8fe1995b-4b6d-4911-b563-a759467fdf53}</Project>
<Name>Clojure ASP.Net MVC Controller Library1</Name>
</ProjectReference>
Visual Studio by default doesn't resolve the project reference correctly, because it assumes there will only be one output, Clojure ASP.Net MVC Controller Library1.dll.
Examples of the actual output files are MvcApplication1.Controllers.HomeController.dll and HomeController.clj
I'd like to make this work without making any changes to the C# .csproj file, so that a .cljproj can easily be referenced from any .csproj file.
My attempt to resolve the project reference is by overriding the GetTargetPath target.
<Target Name="GetTargetPath" DependsOnTargets="$(GetTargetPathDependsOn)" Returns="#(TargetPath)">
<ItemGroup>
<TargetPath Include="$(TargetDir)\**\*.dll" />
<!-- <TargetPath Include="$(TargetDir)\**\*.clj" /> -->
</ItemGroup>
</Target>
If I set the TargetPath using *.dll, it works and copies the .dll files to the c#.csproj output directory. It even copies the .pdb files to that directory, although I didn't add them to the TargetPath. However, if I uncomment the *.clj TargetPath, CSC complains that the .clj files are corrupt (probably because they are plain text, not .net assemblies).
I'm happy to use a copy command instead of overriding TargetPath, however I'm not sure which variable to use for the directory to output them to, because $(outdir) gives me the bin of my custom project (.cljproj) not the bin of the c# project trying to resolve the project reference (.csproj). I'm not sure what other target to override besides GetTargetPath, because most clojure project (.cljproj) targets are not called when compiling the c# (.csjproj) project, for example: .cljproj:AfterBuild is only called when compiling cljproj directly, not when compiling .csproj which has a project reference to .cljproj.
I was able to get it to work by overriding the GetCopyToOutputDirectoryItems Target.
<Target Name="GetCopyToOutputDirectoryItems" Returns="#(CopyToOutputDirectoryItemsWithTargetPath)">
<ItemGroup>
<CopyToOutputDirectoryItems Include="$(TargetDir)\**\*.clj">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</CopyToOutputDirectoryItems>
</ItemGroup>
<AssignTargetPath Files="#(CopyToOutputDirectoryItems)" RootFolder="$(TargetDir)">
<Output TaskParameter="AssignedFiles" ItemName="CopyToOutputDirectoryItemsWithTargetPath" />
</AssignTargetPath>
</Target>

multiple build rules using vc2010 and msbuild

I'm managing a C++ project in VS2010 and want to have ALL .cpp files run through an external tool before going to the C++ compiler. All signs seem to indicate this is possible. See, for example, here.
Since this will happen over multiple projects, it makes sense to put this functionality in a property sheet and then just importing this property sheet everywhere. Before I touched the property sheet, it looked like this in its entirety:
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<_ProjectFileVersion>10.0.30319.1</_ProjectFileVersion>
<ExtensionsToDeleteOnClean>...</ExtensionsToDeleteOnClean>
<CustomBuildBeforeTargets>ClCompile</CustomBuildBeforeTargets>
</PropertyGroup>
<ItemDefinitionGroup>
<ClCompile>
<AdditionalIncludeDirectories>...</AdditionalIncludeDirectories>
<ForcedIncludeFiles>%(ForcedIncludeFiles)</ForcedIncludeFiles>
</ClCompile>
<Link>
<AdditionalDependencies>...</AdditionalDependencies>
</Link>
<Outputs>...</Outputs>
</CustomBuildStep>
</ItemDefinitionGroup>
</Project>
As per the above linked document, I have added the following lines:
<Project>
<PropertyGroup>
...
<CustomBuildBeforeTargets>ClCompile</CustomBuildBeforeTargets>
</PropertyGroup>
...
<ItemGroup>
<CustomBuild Include="*.cpp">
<Message>Running Custom Build Step</Message>
<Command>dummy</Command>
<Outputs>dummy</Outputs>
</CustomBuild>
</ItemGroup>
</Project>
This appears to have no effect, and my custom build tool never runs before ClCompile. I have tried various ways of moving things around and renaming tags (the Xml editor complains that CustomBuild isn't even valid according to the schema, for example), but nothing works.
What am I doing wrong?
Although this is a very old question, I will attempt to answer it.
The Include attribute expects to find *.cpp files in the Project directory. If there are no .cpp files in the Project directory, then this CustomBuild "Task" will never kick in.
What the OP could have done was
<CustomBuild Include="**\*.cpp">
<Message>Custombuild kicking in</Message>
<Command>echo %(Identity)</Command>
<Outputs>dummy</Outputs>
</CustomBuild>
instead of
<CustomBuild Include="*.cpp">
...
and he would have performed the CustomBuild action on all the .cpp files in all the subdirectories of the project directory, and not just the .cpp files waiting to be compiled.
If the .cpp files are not availble under the Project directory (can happen when such a project directory structure is used) then the OP must explicitly point to the right "root" directory.
Alternatively you can save yourself wildcard usage by simply running an Exec task on every included ClCompile.
<Target Name="ProcessClCompileFiles" BeforeTargets="ClCompile" Condition="'#(ClCompile)'!=''">
<Message Text="== Starting processing cpp files ==" Importance="High"/>
<Exec Command="echo processing %(ClCompile.filename)%(ClCompile.extension) & ////YOUR COMMAND HERE////" />
</Target>
As a note. This requires the ClCompile itemgroup to be initialised before this Target. if you were to create a target that dynamically adds ClCompile, you'd have to ensure that this happens before this Target is called.
Another note is that these Exec tasks will always run if there are any items in the ClCompile itemgroup. If you have a condition where it should skip them (eg, already processed), use Condition="A=B"
For easy reference, here's how to validate it hasn't created a file and if it created a file; if the file is out of date:
Condition="!Exists('$(OutDir)%(Filename)%(Extension)') OR ($([System.DateTime]::Parse('%(ClCompile.ModifiedTime)').Ticks) > $([System.IO.File]::GetLastWriteTime('$(OutDir)%(Filename)%(Extension)').Ticks))"

MSBuild: Custom.After.Microsoft.Common.targets for native C++ projects in VS2010

I've read about the use of "Custom.Before.Microsoft.Common.targets" and "Custom.After.Microsoft.Common.targets" in order to execute a custom target before/after every project build and I would like to use this technique in order to change version info while building on our TeamCity build server.
The problem is that although it works for C# projects, it doesn't seem to work for native C++ projects.
After some digging around in the Microsoft.Cpp.targets file I found out that for native C++ projects this seems to be implemented through setting $(ForceImportBeforeCppTargets) and $(ForceImportAfterCppTargets).
I can't seem to find a single piece of information on the web about this technique for native C++ apps though, so I'm asking if I'm looking in the right direction or not.
Any help is appreciated.
For VC++ projects it is a bit different. You define a file to be imported either at the beginning or at the end of the project. To use this approach you need to define values for the properties ForceImportBeforeCppTargets or ForceImportAfterCppTargets. For example if you want a file to be included at the beginning of the project you can pass in the value at the command line. For example I just created a dummy VC++ project named CppTets01. Then I created the two sample files below.
Before.proj
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<Target Name="CustomTargetInBefore" AfterTargets="Build">
<Message Text="From CustomTargetInBefore" Importance="high"/>
</Target>
</Project>
After.proj
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<Target Name="CustomTargetInAfter" AfterTargets="Build">
<Message Text="From CustomTargetInAfter" Importance="high"/>
</Target>
</Project>
Then I executed the following command:
msbuild CppTest01.vcxproj
/p:ForceImportBeforeCppTargets="C:\Temp\_NET\ThrowAway\CppTest01\CppTest01\Before.proj";
ForceImportAfterCppTargets="C:\Temp\_NET\ThrowAway\CppTest01\CppTest01\After.proj"
The result was
C:\Temp_NET\ThrowAway\CppTest01\CppTest01>msbuild CppTest01.vcxproj /p:ForceImportBeforeCppTargets="C:\Temp_NET\ThrowAway\CppTest01\C
ppTest01\Before.proj";ForceImportAfterCppTargets="C:\Temp_NET\ThrowAway\CppTest01\CppTest01\After.proj"
Microsoft (R) Build Engine Version 4.0.30319.1
[Microsoft .NET Framework, Version 4.0.30319.1]
Copyright (C) Microsoft Corporation 2007. All rights reserved.
Build started 10/18/2010 8:32:55 AM.
Project "C:\Temp\_NET\ThrowAway\CppTest01\CppTest01\CppTest01.vcxproj" on node 1 (default targets).
InitializeBuildStatus:
Creating "Debug\CppTest01.unsuccessfulbuild" because "AlwaysCreate" was specified.
ClCompile:
All outputs are up-to-date.
All outputs are up-to-date.
ManifestResourceCompile:
All outputs are up-to-date.
Link:
All outputs are up-to-date.
Manifest:
All outputs are up-to-date.
FinalizeBuildStatus:
Deleting file "Debug\CppTest01.unsuccessfulbuild".
Touching "Debug\CppTest01.lastbuildstate".
CustomTargetInBefore:
From CustomTargetInBefore
CustomTargetInAfter:
From CustomTargetInAfter
Done Building Project "C:\Temp\_NET\ThrowAway\CppTest01\CppTest01\CppTest01.vcxproj" (default targets).
Build succeeded.
0 Warning(s)
0 Error(s)
Time Elapsed 00:00:00.21
As you can see from the output the targets were successfully injected into the build process. If you want to relate this back to Custom.Before.Microsoft.Common.targets and Custom.Before.Microsoft.Common.targets then you should know that the technique used there is a bit different. Specifically if you create those files they are automatically imported into every C#/VB.NET project. In this case you have to set this property. You really have two options here:
You can set this property as an environment variable
You can use another technique, ImportBefore & ImportAfter which is specific to VC++
For #1 let me explain a bit. In MSBuild when you access a property with the syntax $(PropName) then if a property with the name PropName doesn't exist MSBuild will look up in the environment variables to see if such a value exists, if it does then that value is returned. So if you have a build server in which you want to include a file for each VC++ build, then just create those properties as environment variables. Now for the other technique.
ImportBefore/ImportAfter
In VC++ a new concept is introduced. In Microsoft.Cpp.Win32.targets you can see the declaration at the top of the .targets file.
<Import Project="$(VCTargetsPath)\Platforms\Win32\ImportBefore\*.targets"
Condition="Exists('$(VCTargetsPath)\Platforms\Win32\ImportBefore')" />
Then there is one towards the bottom
<Import Project="$(VCTargetsPath)\Platforms\Win32\ImportAfter\*.targets"
Condition="Exists('$(VCTargetsPath)\Platforms\Win32\ImportAfter')" />
A similar import declaration exists for the other target platforms as well. Take a look at the files at %ProgramFiles32%\MSBuild\Microsoft.Cpp\v4.0\Platforms\ for the specific names.
With this technique if you want a file to be imported then simply create a file that ends with .targets and place it into the appropriate folder. The advantage of this is that it will be imported into every VC++ build for that platform, and that you can create many different files. The drawback is that you have to place them in those specific folders. That's the main difference between both techniques. With this first technique you can specify the file location via property and its not automatically included for every build, but for the second approach it is but you cannot change the location
You can also add project content into one of *.props files from directory %LOCALAPPDATA%\Microsoft\MSBuild\v4.0\
It make same effect.

How do I add a custom build target to a Visual C++ 2010 project?

There are plenty of guides out there which help you mimic VS2008's "Custom Build Step" in VS2010 with MSBuild. However, I'd like my build to be smarter and make use of MSBuild. I've written a little MSBuild task which invokes the ANTLR parser generator. That build task works flawlessly when I run it in a simple test MSBuild file. However, when I try to add my task to a C++ project, I run into problems. Essentially I've added this to the top of my project file (Right after the <project> element):
<UsingTask TaskName="ANTLR.MSBuild.AntlrGrammar"
AssemblyName = "ANTLR.MSBuild, Version=1.0.0.0, Culture=neutral, PublicKeyToken=d50cc80512acc876" />
<Target Name="BeforeBuild"
Inputs="ConfigurationParser.g"
Outputs="ConfigurationParserParser.h;ConfigurationParserParser.cpp;ConfigurationParserLexer.h;ConfigurationParserLexer.cpp">
<AntlrGrammar
AntlrLocation="$(MSBuildProjectDirectory)Antlr.jar"
Grammar="ConfigurationParser.g"
RenameToCpp="true" />
</Target>
However, my target is not being called before build.
How can I add my task to a C++ build?
Before reading this answer, you'll probably want to see:
General .vcxproj File Reference
The New Way of doing Build Extensibility in .NET 4
The old way of extending MSBuild, and the one mentioned by the reference book I have, essentially is based on overriding default-empty targets supplied by Microsoft. The new way, as specified in the second link above, is to define your own arbitrary target, and use the "BeforeTargets" and "AfterTargets" properties to force your target to run before or after your intended target.
In my specific case, I needed the ANTLR Grammars task to run before the CLCompile target, which actually builds the C++ files, because the ANTLR Grammars task builds .cpp files. Therefore, the XML looks like this:
<Project ...
<!-- Other things put in by VS2010 ... this is the bottom of the file -->
<UsingTask TaskName="ANTLR.MSBuild.AntlrGrammar"
AssemblyName = "ANTLR.MSBuild, Version=1.0.0.0, Culture=neutral, PublicKeyToken=d50cc80512acc876" />
<Target Name="AntlrGrammars"
Inputs="Configuration.g"
Outputs="ConfigurationParser.h;ConfigurationParser.cpp;ConfigurationLexer.h;ConfigurationLexer.cpp"
BeforeTargets="ClCompile">
<AntlrGrammar
AntlrLocation="$(MSBuildProjectDirectory)\Antlr.jar"
Grammar="Configuration.g"
RenameToCpp="true" />
</Target>
<ImportGroup Label="ExtensionTargets">
</ImportGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
</Project>
As for why this is superior to a PreBuildEvent and/or PostBuildEvent; this is smart enough to not rebuild the .cpps when the grammar itself is not updated. You'll get something like:
1>AntlrGrammars:
1>Skipping target "AntlrGrammars" because all output files are up-to-date with respect to the input files.
1>ClCompile:
1> All outputs are up-to-date.
1> All outputs are up-to-date.
This also silences Visual Studio's incessant complaining every time you run the program that it needs to rebuild things, like it does with plain pre- and post- build steps.
Hope this helps someone -- took me frickin forever to figure out.