Looking for recommendations how to best use Maven in a muti-stage Pipeline build to only deploy at the end

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Looking for recommendations how to best use Maven in a muti-stage Pipeline build to only deploy at the end

Eric B
I'm looking for recommendations for the best way to use Maven in a
multi-stage Jenkins pipeline build to deploy only at the end.  At the
moment, I'm using Sonatype Nexus 3.x, which means i don't have the benefit
of staging repos.   Consequently, I have to ensure that the only released
versions of my libraries/application are final - they've passed QA.
Additionally, the team wants to ensure that the version numbers are always
incremental and every version in the repo is a consumable version (ie: I
cannot deploy a version 1.2.3 which has not passed QA/Acceptance Tests, and
furthermore, I cannot deploy a 1.2.2 followed by a 1.2.4).

What that requirement translates to is that I have to ensure that the
binary built is fully tested before promoting it to Nexus. (and that I
shouldn't be appending build numbers to the maven version number).

In my mind, I would like to do something the following in a Pipeline build:

stage('build') { steps { sh 'mvn clean install'} }

stage('Confirm deploy to QA'){

steps {

checkpoint 'test server deployed'

script {

env.DEPLOY_TO_QA_TEST = input message: 'User input required',

submitter: 'authenticated',

parameters: [choice(name: 'Deploy to acceptance test server', choices:
'no\nyes', description: 'Choose "yes" if you want to deploy the QA test
server')]

}

}

}

stage('deployQA') {

when {

environment name: 'DEPLOY_TO_QA_TEST', value: 'yes'

}

steps{

/* deploy the build to a QA environment */  }

}


stage('Confirm deploy to UAT'){ // prompt user }

stage {'deployUAT') { /* deploy the build to a PreProd/User Acceptance
Testing enviornment */}


stage('Confirm publish to Nexus'){ // prompt user }

stage('publish') {

    steps {

      // mvn deploy -DskipTests (just deploy - don't rebuild)

    }

}


Basically, I want to design my Jenkins pipeline to be my staging process.
The problem I have is I'm not sure how I can only deploy at the end of the
pipeline.  When maven runs the deploy lifecycle, it will run through all
the other stages and reassemble my binaries, which technically are not
longer the same as those that were approved.  So consequently, my binary
hashes that were approved earlier in the pipeline are not the same hashes
that are deployed in Nexus.

I realize that i can probably do some work and use the Reproducible Build
plugin (https://zlika.github.io/reproducible-build-maven-plugin/), but that
too comes with drawbacks (I want build timestamps in my Manifest files, and
zip entries, etc).

Am I sunk?  Is my only hope to wait until Sonatype releases Staging repos
for Nexus 3.x sometime in Q2?  Or is there some other way I can work around
this?

How is everyone else handling this situation?

Thanks for sharing.

Eric
Reply | Threaded
Open this post in threaded view
|

RE: Looking for recommendations how to best use Maven in a muti-stage Pipeline build to only deploy at the end

Robert Patrick
While Nexus may not support "staging repositories", it certainly supports more than one repository so why not just create one or more repositories that serve as staging repositories.  For example,

Pipeline Steps:
1.) Trigger a build based on source check-in and push to stage1 repo if build "succeeds"
2.) Pull artifacts from stage1 repo, run stage 2 tests, and push to stage2 repo if tests succeed.
3.) Pull artifacts from stage2 repo, run stage 3 tests, and push to stage3 repo if tests succeed.
4.) Pull artifacts from stage3 repo, run UAT tests, and push to release repo if tests succeed.




-----Original Message-----
From: Eric B [mailto:[hidden email]]
Sent: Wednesday, February 14, 2018 8:30 AM
To: Maven Users List <[hidden email]>
Subject: Looking for recommendations how to best use Maven in a muti-stage Pipeline build to only deploy at the end

I'm looking for recommendations for the best way to use Maven in a multi-stage Jenkins pipeline build to deploy only at the end.  At the moment, I'm using Sonatype Nexus 3.x, which means i don't have the benefit
of staging repos.   Consequently, I have to ensure that the only released
versions of my libraries/application are final - they've passed QA.
Additionally, the team wants to ensure that the version numbers are always incremental and every version in the repo is a consumable version (ie: I cannot deploy a version 1.2.3 which has not passed QA/Acceptance Tests, and furthermore, I cannot deploy a 1.2.2 followed by a 1.2.4).

What that requirement translates to is that I have to ensure that the binary built is fully tested before promoting it to Nexus. (and that I shouldn't be appending build numbers to the maven version number).

In my mind, I would like to do something the following in a Pipeline build:

stage('build') { steps { sh 'mvn clean install'} }

stage('Confirm deploy to QA'){

steps {

checkpoint 'test server deployed'

script {

env.DEPLOY_TO_QA_TEST = input message: 'User input required',

submitter: 'authenticated',

parameters: [choice(name: 'Deploy to acceptance test server', choices:
'no\nyes', description: 'Choose "yes" if you want to deploy the QA test server')]

}

}

}

stage('deployQA') {

when {

environment name: 'DEPLOY_TO_QA_TEST', value: 'yes'

}

steps{

/* deploy the build to a QA environment */  }

}


stage('Confirm deploy to UAT'){ // prompt user }

stage {'deployUAT') { /* deploy the build to a PreProd/User Acceptance Testing enviornment */}


stage('Confirm publish to Nexus'){ // prompt user }

stage('publish') {

    steps {

      // mvn deploy -DskipTests (just deploy - don't rebuild)

    }

}


Basically, I want to design my Jenkins pipeline to be my staging process.
The problem I have is I'm not sure how I can only deploy at the end of the pipeline.  When maven runs the deploy lifecycle, it will run through all the other stages and reassemble my binaries, which technically are not longer the same as those that were approved.  So consequently, my binary hashes that were approved earlier in the pipeline are not the same hashes that are deployed in Nexus.

I realize that i can probably do some work and use the Reproducible Build plugin (https://urldefense.proofpoint.com/v2/url?u=https-3A__zlika.github.io_reproducible-2Dbuild-2Dmaven-2Dplugin_&d=DwIBaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=nSxyAsyxa1Izff8ULe7vW8u084madbci-hLPsiLpxeU&m=Og2S17jc02JRwm-oHae8UQmiIg_ygbBRL0EQoB_Wvuw&s=zipmYyPLpmFYv1RsnquZtQMf0-HoYoix12SZj6gD2jM&e=), but that too comes with drawbacks (I want build timestamps in my Manifest files, and zip entries, etc).

Am I sunk?  Is my only hope to wait until Sonatype releases Staging repos for Nexus 3.x sometime in Q2?  Or is there some other way I can work around this?

How is everyone else handling this situation?

Thanks for sharing.

Eric

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Looking for recommendations how to best use Maven in a muti-stagePipeline build to only deploy at the end

Bernd Eckenfels
I am not sure why Nexus does not Support staging repos, I thought this is one of its defining Features? Certainly the ASF is using Nexus this way.

Anyway, another solution is to use a Feature of the CI to run a whole Pipeline on a single executor and have this executor have a local maven repo instance (this is a good idea anyway). Then you can avoid to deploy at all (only install) until your last step.

In my cases it Comes up not so often as the last Pipeline steps to release a product depend on Long ago released artifact versions with their own lifecylce.

Gruss
Bernd
--
http://bernd.eckenfels.net

Von: Robert Patrick
Gesendet: Mittwoch, 14. Februar 2018 16:01
An: Maven Users List
Betreff: RE: Looking for recommendations how to best use Maven in a muti-stagePipeline build to only deploy at the end

While Nexus may not support "staging repositories", it certainly supports more than one repository so why not just create one or more repositories that serve as staging repositories.  For example,

Pipeline Steps:
1.) Trigger a build based on source check-in and push to stage1 repo if build "succeeds"
2.) Pull artifacts from stage1 repo, run stage 2 tests, and push to stage2 repo if tests succeed.
3.) Pull artifacts from stage2 repo, run stage 3 tests, and push to stage3 repo if tests succeed.
4.) Pull artifacts from stage3 repo, run UAT tests, and push to release repo if tests succeed.




-----Original Message-----
From: Eric B [mailto:[hidden email]]
Sent: Wednesday, February 14, 2018 8:30 AM
To: Maven Users List <[hidden email]>
Subject: Looking for recommendations how to best use Maven in a muti-stage Pipeline build to only deploy at the end

I'm looking for recommendations for the best way to use Maven in a multi-stage Jenkins pipeline build to deploy only at the end.  At the moment, I'm using Sonatype Nexus 3.x, which means i don't have the benefit
of staging repos.   Consequently, I have to ensure that the only released
versions of my libraries/application are final - they've passed QA.
Additionally, the team wants to ensure that the version numbers are always incremental and every version in the repo is a consumable version (ie: I cannot deploy a version 1.2.3 which has not passed QA/Acceptance Tests, and furthermore, I cannot deploy a 1.2.2 followed by a 1.2.4).

What that requirement translates to is that I have to ensure that the binary built is fully tested before promoting it to Nexus. (and that I shouldn't be appending build numbers to the maven version number).

In my mind, I would like to do something the following in a Pipeline build:

stage('build') { steps { sh 'mvn clean install'} }

stage('Confirm deploy to QA'){

steps {

checkpoint 'test server deployed'

script {

env.DEPLOY_TO_QA_TEST = input message: 'User input required',

submitter: 'authenticated',

parameters: [choice(name: 'Deploy to acceptance test server', choices:
'no\nyes', description: 'Choose "yes" if you want to deploy the QA test server')]

}

}

}

stage('deployQA') {

when {

environment name: 'DEPLOY_TO_QA_TEST', value: 'yes'

}

steps{

/* deploy the build to a QA environment */  }

}


stage('Confirm deploy to UAT'){ // prompt user }

stage {'deployUAT') { /* deploy the build to a PreProd/User Acceptance Testing enviornment */}


stage('Confirm publish to Nexus'){ // prompt user }

stage('publish') {

    steps {

      // mvn deploy -DskipTests (just deploy - don't rebuild)

    }

}


Basically, I want to design my Jenkins pipeline to be my staging process.
The problem I have is I'm not sure how I can only deploy at the end of the pipeline.  When maven runs the deploy lifecycle, it will run through all the other stages and reassemble my binaries, which technically are not longer the same as those that were approved.  So consequently, my binary hashes that were approved earlier in the pipeline are not the same hashes that are deployed in Nexus.

I realize that i can probably do some work and use the Reproducible Build plugin (https://urldefense.proofpoint.com/v2/url?u=https-3A__zlika.github.io_reproducible-2Dbuild-2Dmaven-2Dplugin_&d=DwIBaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=nSxyAsyxa1Izff8ULe7vW8u084madbci-hLPsiLpxeU&m=Og2S17jc02JRwm-oHae8UQmiIg_ygbBRL0EQoB_Wvuw&s=zipmYyPLpmFYv1RsnquZtQMf0-HoYoix12SZj6gD2jM&e=), but that too comes with drawbacks (I want build timestamps in my Manifest files, and zip entries, etc).

Am I sunk?  Is my only hope to wait until Sonatype releases Staging repos for Nexus 3.x sometime in Q2?  Or is there some other way I can work around this?

How is everyone else handling this situation?

Thanks for sharing.

Eric

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Looking for recommendations how to best use Maven in a muti-stage Pipeline build to only deploy at the end

Eric Benzacar
In reply to this post by Robert Patrick
Hi Robert,

Thanks for the suggestions.  To be honest, I never thought of using regular
repositories as staging repos.  That being said, I do see a hiccup with the
option that I'm not sure how to address.  In the case where a build is
rejected during the pipelline stage, how do I delete it?  I haven't found a
plugin that would allow me to delete a build from a repo.  And with a
multi-module project, there are many artifacts that would need to be rolled
back.  I guess I could simply configure the repo to allow re-deployment of
the same artifact to these "stage" repos, but if ever a deployment fails
part way through, I end up with a repo in an unstable manner (ie: there is
no way to ensure that all modules are promoted at the same time).

All that being said, the last question would be how to retrieve and
redeploy these artifacts from one repo to another?  Changing the
distributionManagement settings is simple enough, but how would I instruct
maven to download all the modules from a multi-module build and then deploy
them?  I guess I could hack a profile in the parent pom where the different
modules are defined with a dependency:copy-dependencies goal, but how do I
ensure all artifacts that were just downloaded are then deployed to the
next repo in the stage?  I would think the only way to do this would be
writing my own custom plugin, unless you can think of some other mechanism
to accomplish this?

Thanks,

Eric


On Wed, Feb 14, 2018 at 10:01 AM, Robert Patrick <[hidden email]>
wrote:

> While Nexus may not support "staging repositories", it certainly supports
> more than one repository so why not just create one or more repositories
> that serve as staging repositories.  For example,
>
> Pipeline Steps:
> 1.) Trigger a build based on source check-in and push to stage1 repo if
> build "succeeds"
> 2.) Pull artifacts from stage1 repo, run stage 2 tests, and push to stage2
> repo if tests succeed.
> 3.) Pull artifacts from stage2 repo, run stage 3 tests, and push to stage3
> repo if tests succeed.
> 4.) Pull artifacts from stage3 repo, run UAT tests, and push to release
> repo if tests succeed.
>
>
>
>
> -----Original Message-----
> From: Eric B [mailto:[hidden email]]
> Sent: Wednesday, February 14, 2018 8:30 AM
> To: Maven Users List <[hidden email]>
> Subject: Looking for recommendations how to best use Maven in a muti-stage
> Pipeline build to only deploy at the end
>
> I'm looking for recommendations for the best way to use Maven in a
> multi-stage Jenkins pipeline build to deploy only at the end.  At the
> moment, I'm using Sonatype Nexus 3.x, which means i don't have the benefit
> of staging repos.   Consequently, I have to ensure that the only released
> versions of my libraries/application are final - they've passed QA.
> Additionally, the team wants to ensure that the version numbers are always
> incremental and every version in the repo is a consumable version (ie: I
> cannot deploy a version 1.2.3 which has not passed QA/Acceptance Tests, and
> furthermore, I cannot deploy a 1.2.2 followed by a 1.2.4).
>
> What that requirement translates to is that I have to ensure that the
> binary built is fully tested before promoting it to Nexus. (and that I
> shouldn't be appending build numbers to the maven version number).
>
> In my mind, I would like to do something the following in a Pipeline build:
>
> stage('build') { steps { sh 'mvn clean install'} }
>
> stage('Confirm deploy to QA'){
>
> steps {
>
> checkpoint 'test server deployed'
>
> script {
>
> env.DEPLOY_TO_QA_TEST = input message: 'User input required',
>
> submitter: 'authenticated',
>
> parameters: [choice(name: 'Deploy to acceptance test server', choices:
> 'no\nyes', description: 'Choose "yes" if you want to deploy the QA test
> server')]
>
> }
>
> }
>
> }
>
> stage('deployQA') {
>
> when {
>
> environment name: 'DEPLOY_TO_QA_TEST', value: 'yes'
>
> }
>
> steps{
>
> /* deploy the build to a QA environment */  }
>
> }
>
>
> stage('Confirm deploy to UAT'){ // prompt user }
>
> stage {'deployUAT') { /* deploy the build to a PreProd/User Acceptance
> Testing enviornment */}
>
>
> stage('Confirm publish to Nexus'){ // prompt user }
>
> stage('publish') {
>
>     steps {
>
>       // mvn deploy -DskipTests (just deploy - don't rebuild)
>
>     }
>
> }
>
>
> Basically, I want to design my Jenkins pipeline to be my staging process.
> The problem I have is I'm not sure how I can only deploy at the end of the
> pipeline.  When maven runs the deploy lifecycle, it will run through all
> the other stages and reassemble my binaries, which technically are not
> longer the same as those that were approved.  So consequently, my binary
> hashes that were approved earlier in the pipeline are not the same hashes
> that are deployed in Nexus.
>
> I realize that i can probably do some work and use the Reproducible Build
> plugin (https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__zlika.github.io_reproducible-2Dbuild-2Dmaven-2Dplugin_&d=DwIBaQ&c=
> RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=
> nSxyAsyxa1Izff8ULe7vW8u084madbci-hLPsiLpxeU&m=Og2S17jc02JRwm-oHae8UQmiIg_
> ygbBRL0EQoB_Wvuw&s=zipmYyPLpmFYv1RsnquZtQMf0-HoYoix12SZj6gD2jM&e=), but
> that too comes with drawbacks (I want build timestamps in my Manifest
> files, and zip entries, etc).
>
> Am I sunk?  Is my only hope to wait until Sonatype releases Staging repos
> for Nexus 3.x sometime in Q2?  Or is there some other way I can work around
> this?
>
> How is everyone else handling this situation?
>
> Thanks for sharing.
>
> Eric
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Looking for recommendations how to best use Maven in a muti-stage Pipeline build to only deploy at the end

Eric B
In reply to this post by Robert Patrick
Hi Robert,

Thanks for the suggestions.  To be honest, I never thought of using regular
repositories as staging repos.  That being said, I do see a hiccup with the
option that I'm not sure how to address.  In the case where a build is
rejected during the pipelline stage, how do I delete it?  I haven't found a
plugin that would allow me to delete a build from a repo.  And with a
multi-module project, there are many artifacts that would need to be rolled
back.  I guess I could simply configure the repo to allow re-deployment of
the same artifact to these "stage" repos, but if ever a deployment fails
part way through, I end up with a repo in an unstable manner (ie: there is
no way to ensure that all modules are promoted at the same time).

All that being said, the last question would be how to retrieve and
redeploy these artifacts from one repo to another?  Changing the
distributionManagement settings is simple enough, but how would I instruct
maven to download all the modules from a multi-module build and then deploy
them?  I guess I could hack a profile in the parent pom where the different
modules are defined with a dependency:copy-dependencies goal, but how do I
ensure all artifacts that were just downloaded are then deployed to the
next repo in the stage?  I would think the only way to do this would be
writing my own custom plugin, unless you can think of some other mechanism
to accomplish this?

Thanks,

Eric



On Wed, Feb 14, 2018 at 10:01 AM, Robert Patrick <[hidden email]>
wrote:

> While Nexus may not support "staging repositories", it certainly supports
> more than one repository so why not just create one or more repositories
> that serve as staging repositories.  For example,
>
> Pipeline Steps:
> 1.) Trigger a build based on source check-in and push to stage1 repo if
> build "succeeds"
> 2.) Pull artifacts from stage1 repo, run stage 2 tests, and push to stage2
> repo if tests succeed.
> 3.) Pull artifacts from stage2 repo, run stage 3 tests, and push to stage3
> repo if tests succeed.
> 4.) Pull artifacts from stage3 repo, run UAT tests, and push to release
> repo if tests succeed.
>
>
>
>
> -----Original Message-----
> From: Eric B [mailto:[hidden email]]
> Sent: Wednesday, February 14, 2018 8:30 AM
> To: Maven Users List <[hidden email]>
> Subject: Looking for recommendations how to best use Maven in a muti-stage
> Pipeline build to only deploy at the end
>
> I'm looking for recommendations for the best way to use Maven in a
> multi-stage Jenkins pipeline build to deploy only at the end.  At the
> moment, I'm using Sonatype Nexus 3.x, which means i don't have the benefit
> of staging repos.   Consequently, I have to ensure that the only released
> versions of my libraries/application are final - they've passed QA.
> Additionally, the team wants to ensure that the version numbers are always
> incremental and every version in the repo is a consumable version (ie: I
> cannot deploy a version 1.2.3 which has not passed QA/Acceptance Tests, and
> furthermore, I cannot deploy a 1.2.2 followed by a 1.2.4).
>
> What that requirement translates to is that I have to ensure that the
> binary built is fully tested before promoting it to Nexus. (and that I
> shouldn't be appending build numbers to the maven version number).
>
> In my mind, I would like to do something the following in a Pipeline build:
>
> stage('build') { steps { sh 'mvn clean install'} }
>
> stage('Confirm deploy to QA'){
>
> steps {
>
> checkpoint 'test server deployed'
>
> script {
>
> env.DEPLOY_TO_QA_TEST = input message: 'User input required',
>
> submitter: 'authenticated',
>
> parameters: [choice(name: 'Deploy to acceptance test server', choices:
> 'no\nyes', description: 'Choose "yes" if you want to deploy the QA test
> server')]
>
> }
>
> }
>
> }
>
> stage('deployQA') {
>
> when {
>
> environment name: 'DEPLOY_TO_QA_TEST', value: 'yes'
>
> }
>
> steps{
>
> /* deploy the build to a QA environment */  }
>
> }
>
>
> stage('Confirm deploy to UAT'){ // prompt user }
>
> stage {'deployUAT') { /* deploy the build to a PreProd/User Acceptance
> Testing enviornment */}
>
>
> stage('Confirm publish to Nexus'){ // prompt user }
>
> stage('publish') {
>
>     steps {
>
>       // mvn deploy -DskipTests (just deploy - don't rebuild)
>
>     }
>
> }
>
>
> Basically, I want to design my Jenkins pipeline to be my staging process.
> The problem I have is I'm not sure how I can only deploy at the end of the
> pipeline.  When maven runs the deploy lifecycle, it will run through all
> the other stages and reassemble my binaries, which technically are not
> longer the same as those that were approved.  So consequently, my binary
> hashes that were approved earlier in the pipeline are not the same hashes
> that are deployed in Nexus.
>
> I realize that i can probably do some work and use the Reproducible Build
> plugin (https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__zlika.github.io_reproducible-2Dbuild-2Dmaven-2Dplugin_&d=DwIBaQ&c=
> RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=
> nSxyAsyxa1Izff8ULe7vW8u084madbci-hLPsiLpxeU&m=Og2S17jc02JRwm-oHae8UQmiIg_
> ygbBRL0EQoB_Wvuw&s=zipmYyPLpmFYv1RsnquZtQMf0-HoYoix12SZj6gD2jM&e=), but
> that too comes with drawbacks (I want build timestamps in my Manifest
> files, and zip entries, etc).
>
> Am I sunk?  Is my only hope to wait until Sonatype releases Staging repos
> for Nexus 3.x sometime in Q2?  Or is there some other way I can work around
> this?
>
> How is everyone else handling this situation?
>
> Thanks for sharing.
>
> Eric
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Looking for recommendations how to best use Maven in a muti-stagePipeline build to only deploy at the end

Eric B
In reply to this post by Bernd Eckenfels
Bernd,

Nexus 3.x does not support staging repos b/c they are rewriting the entire
platform to support not just Maven artifacts, but any type of repo-based
artifact.  Ex: docker images, npm dependencies, etc...  But the new feature
won't be available until at least April or May.

With respect to running the whole Pipeline on a single executor with a
local maven repo was what I was thinking of doing.  But I still don't know
how to get it to deploy only at the last step without rebuilding again.  If
my last step just calls the deploy goal, then it will still rebuild the
entire artifact - not just grab whatever is in the local repo and deploy
it.  And I'm looking to avoid rebuilding the artifact since I want the
original artifact (the one sitting in the local m2 folder) to be the one
pushed to the repo.

Any suggestions?

Thanks,

Eric


On Wed, Feb 14, 2018 at 10:37 AM, Bernd Eckenfels <[hidden email]>
wrote:

> I am not sure why Nexus does not Support staging repos, I thought this is
> one of its defining Features? Certainly the ASF is using Nexus this way.
>
> Anyway, another solution is to use a Feature of the CI to run a whole
> Pipeline on a single executor and have this executor have a local maven
> repo instance (this is a good idea anyway). Then you can avoid to deploy at
> all (only install) until your last step.
>
> In my cases it Comes up not so often as the last Pipeline steps to release
> a product depend on Long ago released artifact versions with their own
> lifecylce.
>
> Gruss
> Bernd
> --
> http://bernd.eckenfels.net
>
> Von: Robert Patrick
> Gesendet: Mittwoch, 14. Februar 2018 16:01
> An: Maven Users List
> Betreff: RE: Looking for recommendations how to best use Maven in a
> muti-stagePipeline build to only deploy at the end
>
> While Nexus may not support "staging repositories", it certainly supports
> more than one repository so why not just create one or more repositories
> that serve as staging repositories.  For example,
>
> Pipeline Steps:
> 1.) Trigger a build based on source check-in and push to stage1 repo if
> build "succeeds"
> 2.) Pull artifacts from stage1 repo, run stage 2 tests, and push to stage2
> repo if tests succeed.
> 3.) Pull artifacts from stage2 repo, run stage 3 tests, and push to stage3
> repo if tests succeed.
> 4.) Pull artifacts from stage3 repo, run UAT tests, and push to release
> repo if tests succeed.
>
>
>
>
> -----Original Message-----
> From: Eric B [mailto:[hidden email]]
> Sent: Wednesday, February 14, 2018 8:30 AM
> To: Maven Users List <[hidden email]>
> Subject: Looking for recommendations how to best use Maven in a muti-stage
> Pipeline build to only deploy at the end
>
> I'm looking for recommendations for the best way to use Maven in a
> multi-stage Jenkins pipeline build to deploy only at the end.  At the
> moment, I'm using Sonatype Nexus 3.x, which means i don't have the benefit
> of staging repos.   Consequently, I have to ensure that the only released
> versions of my libraries/application are final - they've passed QA.
> Additionally, the team wants to ensure that the version numbers are always
> incremental and every version in the repo is a consumable version (ie: I
> cannot deploy a version 1.2.3 which has not passed QA/Acceptance Tests, and
> furthermore, I cannot deploy a 1.2.2 followed by a 1.2.4).
>
> What that requirement translates to is that I have to ensure that the
> binary built is fully tested before promoting it to Nexus. (and that I
> shouldn't be appending build numbers to the maven version number).
>
> In my mind, I would like to do something the following in a Pipeline build:
>
> stage('build') { steps { sh 'mvn clean install'} }
>
> stage('Confirm deploy to QA'){
>
> steps {
>
> checkpoint 'test server deployed'
>
> script {
>
> env.DEPLOY_TO_QA_TEST = input message: 'User input required',
>
> submitter: 'authenticated',
>
> parameters: [choice(name: 'Deploy to acceptance test server', choices:
> 'no\nyes', description: 'Choose "yes" if you want to deploy the QA test
> server')]
>
> }
>
> }
>
> }
>
> stage('deployQA') {
>
> when {
>
> environment name: 'DEPLOY_TO_QA_TEST', value: 'yes'
>
> }
>
> steps{
>
> /* deploy the build to a QA environment */  }
>
> }
>
>
> stage('Confirm deploy to UAT'){ // prompt user }
>
> stage {'deployUAT') { /* deploy the build to a PreProd/User Acceptance
> Testing enviornment */}
>
>
> stage('Confirm publish to Nexus'){ // prompt user }
>
> stage('publish') {
>
>     steps {
>
>       // mvn deploy -DskipTests (just deploy - don't rebuild)
>
>     }
>
> }
>
>
> Basically, I want to design my Jenkins pipeline to be my staging process.
> The problem I have is I'm not sure how I can only deploy at the end of the
> pipeline.  When maven runs the deploy lifecycle, it will run through all
> the other stages and reassemble my binaries, which technically are not
> longer the same as those that were approved.  So consequently, my binary
> hashes that were approved earlier in the pipeline are not the same hashes
> that are deployed in Nexus.
>
> I realize that i can probably do some work and use the Reproducible Build
> plugin (https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__zlika.github.io_reproducible-2Dbuild-2Dmaven-2Dplugin_&d=DwIBaQ&c=
> RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=
> nSxyAsyxa1Izff8ULe7vW8u084madbci-hLPsiLpxeU&m=Og2S17jc02JRwm-oHae8UQmiIg_
> ygbBRL0EQoB_Wvuw&s=zipmYyPLpmFYv1RsnquZtQMf0-HoYoix12SZj6gD2jM&e=), but
> that too comes with drawbacks (I want build timestamps in my Manifest
> files, and zip entries, etc).
>
> Am I sunk?  Is my only hope to wait until Sonatype releases Staging repos
> for Nexus 3.x sometime in Q2?  Or is there some other way I can work around
> this?
>
> How is everyone else handling this situation?
>
> Thanks for sharing.
>
> Eric
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
>
Reply | Threaded
Open this post in threaded view
|

RE: Looking for recommendations how to best use Maven in a muti-stage Pipeline build to only deploy at the end

Robert Patrick
In reply to this post by Eric B
Inline...


-----Original Message-----
From: Eric B [mailto:[hidden email]]
Sent: Wednesday, February 14, 2018 8:27 PM
To: Maven Users List <[hidden email]>
Subject: Re: Looking for recommendations how to best use Maven in a muti-stage Pipeline build to only deploy at the end

Hi Robert,

Thanks for the suggestions.  To be honest, I never thought of using regular repositories as staging repos.  That being said, I do see a hiccup with the option that I'm not sure how to address.  In the case where a build is rejected during the pipelline stage, how do I delete it?  I haven't found a plugin that would allow me to delete a build from a repo.  And with a multi-module project, there are many artifacts that would need to be rolled back.  I guess I could simply configure the repo to allow re-deployment of the same artifact to these "stage" repos, but if ever a deployment fails part way through, I end up with a repo in an unstable manner (ie: there is no way to ensure that all modules are promoted at the same time).

[Robert] Typically, I would solve this problem by simply putting expiration dates on artifacts in the staging repositories.  I have never used Nexus but Artifactory allows you to limit artifacts in the repository by size, last time accessed, etc.  No need to keep artifacts in the "staging repositories" for more than say, one month, anyway since they were either good (and promoted all the way to the release repository) or bad (and are not interesting after they fail).

All that being said, the last question would be how to retrieve and redeploy these artifacts from one repo to another?  Changing the distributionManagement settings is simple enough, but how would I instruct maven to download all the modules from a multi-module build and then deploy them?  I guess I could hack a profile in the parent pom where the different modules are defined with a dependency:copy-dependencies goal, but how do I ensure all artifacts that were just downloaded are then deployed to the next repo in the stage?  I would think the only way to do this would be writing my own custom plugin, unless you can think of some other mechanism to accomplish this?

[Robert] Another way to do this is to use properties for the key values for the repository URLs in the <repositories> and <distributionManagement> sections of the POM.  You simply set a default value for the properties in the root POM (e.g., with appropriate values for the repositories used by the developers) and then override the property values with Maven command-line -Ds (or profiles) in the Jenkins build steps to handle pulling and promoting to the right places...

Thanks,

Eric



On Wed, Feb 14, 2018 at 10:01 AM, Robert Patrick <[hidden email]>
wrote:

> While Nexus may not support "staging repositories", it certainly
> supports more than one repository so why not just create one or more
> repositories that serve as staging repositories.  For example,
>
> Pipeline Steps:
> 1.) Trigger a build based on source check-in and push to stage1 repo
> if build "succeeds"
> 2.) Pull artifacts from stage1 repo, run stage 2 tests, and push to
> stage2 repo if tests succeed.
> 3.) Pull artifacts from stage2 repo, run stage 3 tests, and push to
> stage3 repo if tests succeed.
> 4.) Pull artifacts from stage3 repo, run UAT tests, and push to
> release repo if tests succeed.
>
>
>
>
> -----Original Message-----
> From: Eric B [mailto:[hidden email]]
> Sent: Wednesday, February 14, 2018 8:30 AM
> To: Maven Users List <[hidden email]>
> Subject: Looking for recommendations how to best use Maven in a
> muti-stage Pipeline build to only deploy at the end
>
> I'm looking for recommendations for the best way to use Maven in a
> multi-stage Jenkins pipeline build to deploy only at the end.  At the
> moment, I'm using Sonatype Nexus 3.x, which means i don't have the benefit
> of staging repos.   Consequently, I have to ensure that the only released
> versions of my libraries/application are final - they've passed QA.
> Additionally, the team wants to ensure that the version numbers are
> always incremental and every version in the repo is a consumable
> version (ie: I cannot deploy a version 1.2.3 which has not passed
> QA/Acceptance Tests, and furthermore, I cannot deploy a 1.2.2 followed by a 1.2.4).
>
> What that requirement translates to is that I have to ensure that the
> binary built is fully tested before promoting it to Nexus. (and that I
> shouldn't be appending build numbers to the maven version number).
>
> In my mind, I would like to do something the following in a Pipeline build:
>
> stage('build') { steps { sh 'mvn clean install'} }
>
> stage('Confirm deploy to QA'){
>
> steps {
>
> checkpoint 'test server deployed'
>
> script {
>
> env.DEPLOY_TO_QA_TEST = input message: 'User input required',
>
> submitter: 'authenticated',
>
> parameters: [choice(name: 'Deploy to acceptance test server', choices:
> 'no\nyes', description: 'Choose "yes" if you want to deploy the QA
> test server')]
>
> }
>
> }
>
> }
>
> stage('deployQA') {
>
> when {
>
> environment name: 'DEPLOY_TO_QA_TEST', value: 'yes'
>
> }
>
> steps{
>
> /* deploy the build to a QA environment */  }
>
> }
>
>
> stage('Confirm deploy to UAT'){ // prompt user }
>
> stage {'deployUAT') { /* deploy the build to a PreProd/User Acceptance
> Testing enviornment */}
>
>
> stage('Confirm publish to Nexus'){ // prompt user }
>
> stage('publish') {
>
>     steps {
>
>       // mvn deploy -DskipTests (just deploy - don't rebuild)
>
>     }
>
> }
>
>
> Basically, I want to design my Jenkins pipeline to be my staging process.
> The problem I have is I'm not sure how I can only deploy at the end of
> the pipeline.  When maven runs the deploy lifecycle, it will run
> through all the other stages and reassemble my binaries, which
> technically are not longer the same as those that were approved.  So
> consequently, my binary hashes that were approved earlier in the
> pipeline are not the same hashes that are deployed in Nexus.
>
> I realize that i can probably do some work and use the Reproducible
> Build plugin (https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__zlika.github.io_reproducible-2Dbuild-2Dmaven-2Dplugin_&d=DwIBaQ&c=
> RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=
> nSxyAsyxa1Izff8ULe7vW8u084madbci-hLPsiLpxeU&m=Og2S17jc02JRwm-oHae8UQmi
> Ig_
> ygbBRL0EQoB_Wvuw&s=zipmYyPLpmFYv1RsnquZtQMf0-HoYoix12SZj6gD2jM&e=),
> but that too comes with drawbacks (I want build timestamps in my
> Manifest files, and zip entries, etc).
>
> Am I sunk?  Is my only hope to wait until Sonatype releases Staging
> repos for Nexus 3.x sometime in Q2?  Or is there some other way I can
> work around this?
>
> How is everyone else handling this situation?
>
> Thanks for sharing.
>
> Eric
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Looking for recommendations how to best use Maven in a muti-stage Pipeline build to only deploy at the end

Eric Benzacar
Robert,

I think you misunderstood my problem.  It isn't a problem of defining
different distributionManagement repos; like you said that is fairly easily
handled using properties or -D parameters.

It is more the question of what kind of plugin is required to run which
says:
- download all the artifacts that this multi-module project previously built
- deploy all the artifacts that were just downloaded

I cannot think of how to design/configure a project to accomplish something
like that.  I'm guessing I would have to create a profile which explicitly
declares each artifact produced by the multi-module project in order to do
a dependency:copy, and then similarly, use a build-helper:attach all the
artifacts that were just downloaded.  But that seems like a painful
solution.  Am hoping there is something easier?

Thanks,

Eric


On Wed, Feb 14, 2018 at 9:45 PM, Robert Patrick <[hidden email]>
wrote:

> Inline...
>
>
> -----Original Message-----
> From: Eric B [mailto:[hidden email]]
> Sent: Wednesday, February 14, 2018 8:27 PM
> To: Maven Users List <[hidden email]>
> Subject: Re: Looking for recommendations how to best use Maven in a
> muti-stage Pipeline build to only deploy at the end
>
> Hi Robert,
>
> Thanks for the suggestions.  To be honest, I never thought of using
> regular repositories as staging repos.  That being said, I do see a hiccup
> with the option that I'm not sure how to address.  In the case where a
> build is rejected during the pipelline stage, how do I delete it?  I
> haven't found a plugin that would allow me to delete a build from a repo.
> And with a multi-module project, there are many artifacts that would need
> to be rolled back.  I guess I could simply configure the repo to allow
> re-deployment of the same artifact to these "stage" repos, but if ever a
> deployment fails part way through, I end up with a repo in an unstable
> manner (ie: there is no way to ensure that all modules are promoted at the
> same time).
>
> [Robert] Typically, I would solve this problem by simply putting
> expiration dates on artifacts in the staging repositories.  I have never
> used Nexus but Artifactory allows you to limit artifacts in the repository
> by size, last time accessed, etc.  No need to keep artifacts in the
> "staging repositories" for more than say, one month, anyway since they were
> either good (and promoted all the way to the release repository) or bad
> (and are not interesting after they fail).
>
> All that being said, the last question would be how to retrieve and
> redeploy these artifacts from one repo to another?  Changing the
> distributionManagement settings is simple enough, but how would I instruct
> maven to download all the modules from a multi-module build and then deploy
> them?  I guess I could hack a profile in the parent pom where the different
> modules are defined with a dependency:copy-dependencies goal, but how do I
> ensure all artifacts that were just downloaded are then deployed to the
> next repo in the stage?  I would think the only way to do this would be
> writing my own custom plugin, unless you can think of some other mechanism
> to accomplish this?
>
> [Robert] Another way to do this is to use properties for the key values
> for the repository URLs in the <repositories> and <distributionManagement>
> sections of the POM.  You simply set a default value for the properties in
> the root POM (e.g., with appropriate values for the repositories used by
> the developers) and then override the property values with Maven
> command-line -Ds (or profiles) in the Jenkins build steps to handle pulling
> and promoting to the right places...
>
> Thanks,
>
> Eric
>
>
>
> On Wed, Feb 14, 2018 at 10:01 AM, Robert Patrick <
> [hidden email]>
> wrote:
>
> > While Nexus may not support "staging repositories", it certainly
> > supports more than one repository so why not just create one or more
> > repositories that serve as staging repositories.  For example,
> >
> > Pipeline Steps:
> > 1.) Trigger a build based on source check-in and push to stage1 repo
> > if build "succeeds"
> > 2.) Pull artifacts from stage1 repo, run stage 2 tests, and push to
> > stage2 repo if tests succeed.
> > 3.) Pull artifacts from stage2 repo, run stage 3 tests, and push to
> > stage3 repo if tests succeed.
> > 4.) Pull artifacts from stage3 repo, run UAT tests, and push to
> > release repo if tests succeed.
> >
> >
> >
> >
> > -----Original Message-----
> > From: Eric B [mailto:[hidden email]]
> > Sent: Wednesday, February 14, 2018 8:30 AM
> > To: Maven Users List <[hidden email]>
> > Subject: Looking for recommendations how to best use Maven in a
> > muti-stage Pipeline build to only deploy at the end
> >
> > I'm looking for recommendations for the best way to use Maven in a
> > multi-stage Jenkins pipeline build to deploy only at the end.  At the
> > moment, I'm using Sonatype Nexus 3.x, which means i don't have the
> benefit
> > of staging repos.   Consequently, I have to ensure that the only released
> > versions of my libraries/application are final - they've passed QA.
> > Additionally, the team wants to ensure that the version numbers are
> > always incremental and every version in the repo is a consumable
> > version (ie: I cannot deploy a version 1.2.3 which has not passed
> > QA/Acceptance Tests, and furthermore, I cannot deploy a 1.2.2 followed
> by a 1.2.4).
> >
> > What that requirement translates to is that I have to ensure that the
> > binary built is fully tested before promoting it to Nexus. (and that I
> > shouldn't be appending build numbers to the maven version number).
> >
> > In my mind, I would like to do something the following in a Pipeline
> build:
> >
> > stage('build') { steps { sh 'mvn clean install'} }
> >
> > stage('Confirm deploy to QA'){
> >
> > steps {
> >
> > checkpoint 'test server deployed'
> >
> > script {
> >
> > env.DEPLOY_TO_QA_TEST = input message: 'User input required',
> >
> > submitter: 'authenticated',
> >
> > parameters: [choice(name: 'Deploy to acceptance test server', choices:
> > 'no\nyes', description: 'Choose "yes" if you want to deploy the QA
> > test server')]
> >
> > }
> >
> > }
> >
> > }
> >
> > stage('deployQA') {
> >
> > when {
> >
> > environment name: 'DEPLOY_TO_QA_TEST', value: 'yes'
> >
> > }
> >
> > steps{
> >
> > /* deploy the build to a QA environment */  }
> >
> > }
> >
> >
> > stage('Confirm deploy to UAT'){ // prompt user }
> >
> > stage {'deployUAT') { /* deploy the build to a PreProd/User Acceptance
> > Testing enviornment */}
> >
> >
> > stage('Confirm publish to Nexus'){ // prompt user }
> >
> > stage('publish') {
> >
> >     steps {
> >
> >       // mvn deploy -DskipTests (just deploy - don't rebuild)
> >
> >     }
> >
> > }
> >
> >
> > Basically, I want to design my Jenkins pipeline to be my staging process.
> > The problem I have is I'm not sure how I can only deploy at the end of
> > the pipeline.  When maven runs the deploy lifecycle, it will run
> > through all the other stages and reassemble my binaries, which
> > technically are not longer the same as those that were approved.  So
> > consequently, my binary hashes that were approved earlier in the
> > pipeline are not the same hashes that are deployed in Nexus.
> >
> > I realize that i can probably do some work and use the Reproducible
> > Build plugin (https://urldefense.proofpoint.com/v2/url?u=https-
> > 3A__zlika.github.io_reproducible-2Dbuild-2Dmaven-2Dplugin_&d=DwIBaQ&c=
> > RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=
> > nSxyAsyxa1Izff8ULe7vW8u084madbci-hLPsiLpxeU&m=Og2S17jc02JRwm-oHae8UQmi
> > Ig_
> > ygbBRL0EQoB_Wvuw&s=zipmYyPLpmFYv1RsnquZtQMf0-HoYoix12SZj6gD2jM&e=),
> > but that too comes with drawbacks (I want build timestamps in my
> > Manifest files, and zip entries, etc).
> >
> > Am I sunk?  Is my only hope to wait until Sonatype releases Staging
> > repos for Nexus 3.x sometime in Q2?  Or is there some other way I can
> > work around this?
> >
> > How is everyone else handling this situation?
> >
> > Thanks for sharing.
> >
> > Eric
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Looking for recommendations how to best use Maven in a muti-stage Pipeline build to only deploy at the end

Chas Honton
If you have a multi-module project and the tests are run as part of the last module, you can use the deployAtEnd parameter to delay deploying of artifacts until the end of the build.

chas

> On Feb 14, 2018, at 7:15 PM, Eric Benzacar <[hidden email]> wrote:
>
> Robert,
>
> I think you misunderstood my problem.  It isn't a problem of defining
> different distributionManagement repos; like you said that is fairly easily
> handled using properties or -D parameters.
>
> It is more the question of what kind of plugin is required to run which
> says:
> - download all the artifacts that this multi-module project previously built
> - deploy all the artifacts that were just downloaded
>
> I cannot think of how to design/configure a project to accomplish something
> like that.  I'm guessing I would have to create a profile which explicitly
> declares each artifact produced by the multi-module project in order to do
> a dependency:copy, and then similarly, use a build-helper:attach all the
> artifacts that were just downloaded.  But that seems like a painful
> solution.  Am hoping there is something easier?
>
> Thanks,
>
> Eric
>
>
> On Wed, Feb 14, 2018 at 9:45 PM, Robert Patrick <[hidden email]>
> wrote:
>
>> Inline...
>>
>>
>> -----Original Message-----
>> From: Eric B [mailto:[hidden email]]
>> Sent: Wednesday, February 14, 2018 8:27 PM
>> To: Maven Users List <[hidden email]>
>> Subject: Re: Looking for recommendations how to best use Maven in a
>> muti-stage Pipeline build to only deploy at the end
>>
>> Hi Robert,
>>
>> Thanks for the suggestions.  To be honest, I never thought of using
>> regular repositories as staging repos.  That being said, I do see a hiccup
>> with the option that I'm not sure how to address.  In the case where a
>> build is rejected during the pipelline stage, how do I delete it?  I
>> haven't found a plugin that would allow me to delete a build from a repo.
>> And with a multi-module project, there are many artifacts that would need
>> to be rolled back.  I guess I could simply configure the repo to allow
>> re-deployment of the same artifact to these "stage" repos, but if ever a
>> deployment fails part way through, I end up with a repo in an unstable
>> manner (ie: there is no way to ensure that all modules are promoted at the
>> same time).
>>
>> [Robert] Typically, I would solve this problem by simply putting
>> expiration dates on artifacts in the staging repositories.  I have never
>> used Nexus but Artifactory allows you to limit artifacts in the repository
>> by size, last time accessed, etc.  No need to keep artifacts in the
>> "staging repositories" for more than say, one month, anyway since they were
>> either good (and promoted all the way to the release repository) or bad
>> (and are not interesting after they fail).
>>
>> All that being said, the last question would be how to retrieve and
>> redeploy these artifacts from one repo to another?  Changing the
>> distributionManagement settings is simple enough, but how would I instruct
>> maven to download all the modules from a multi-module build and then deploy
>> them?  I guess I could hack a profile in the parent pom where the different
>> modules are defined with a dependency:copy-dependencies goal, but how do I
>> ensure all artifacts that were just downloaded are then deployed to the
>> next repo in the stage?  I would think the only way to do this would be
>> writing my own custom plugin, unless you can think of some other mechanism
>> to accomplish this?
>>
>> [Robert] Another way to do this is to use properties for the key values
>> for the repository URLs in the <repositories> and <distributionManagement>
>> sections of the POM.  You simply set a default value for the properties in
>> the root POM (e.g., with appropriate values for the repositories used by
>> the developers) and then override the property values with Maven
>> command-line -Ds (or profiles) in the Jenkins build steps to handle pulling
>> and promoting to the right places...
>>
>> Thanks,
>>
>> Eric
>>
>>
>>
>> On Wed, Feb 14, 2018 at 10:01 AM, Robert Patrick <
>> [hidden email]>
>> wrote:
>>
>>> While Nexus may not support "staging repositories", it certainly
>>> supports more than one repository so why not just create one or more
>>> repositories that serve as staging repositories.  For example,
>>>
>>> Pipeline Steps:
>>> 1.) Trigger a build based on source check-in and push to stage1 repo
>>> if build "succeeds"
>>> 2.) Pull artifacts from stage1 repo, run stage 2 tests, and push to
>>> stage2 repo if tests succeed.
>>> 3.) Pull artifacts from stage2 repo, run stage 3 tests, and push to
>>> stage3 repo if tests succeed.
>>> 4.) Pull artifacts from stage3 repo, run UAT tests, and push to
>>> release repo if tests succeed.
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Eric B [mailto:[hidden email]]
>>> Sent: Wednesday, February 14, 2018 8:30 AM
>>> To: Maven Users List <[hidden email]>
>>> Subject: Looking for recommendations how to best use Maven in a
>>> muti-stage Pipeline build to only deploy at the end
>>>
>>> I'm looking for recommendations for the best way to use Maven in a
>>> multi-stage Jenkins pipeline build to deploy only at the end.  At the
>>> moment, I'm using Sonatype Nexus 3.x, which means i don't have the
>> benefit
>>> of staging repos.   Consequently, I have to ensure that the only released
>>> versions of my libraries/application are final - they've passed QA.
>>> Additionally, the team wants to ensure that the version numbers are
>>> always incremental and every version in the repo is a consumable
>>> version (ie: I cannot deploy a version 1.2.3 which has not passed
>>> QA/Acceptance Tests, and furthermore, I cannot deploy a 1.2.2 followed
>> by a 1.2.4).
>>>
>>> What that requirement translates to is that I have to ensure that the
>>> binary built is fully tested before promoting it to Nexus. (and that I
>>> shouldn't be appending build numbers to the maven version number).
>>>
>>> In my mind, I would like to do something the following in a Pipeline
>> build:
>>>
>>> stage('build') { steps { sh 'mvn clean install'} }
>>>
>>> stage('Confirm deploy to QA'){
>>>
>>> steps {
>>>
>>> checkpoint 'test server deployed'
>>>
>>> script {
>>>
>>> env.DEPLOY_TO_QA_TEST = input message: 'User input required',
>>>
>>> submitter: 'authenticated',
>>>
>>> parameters: [choice(name: 'Deploy to acceptance test server', choices:
>>> 'no\nyes', description: 'Choose "yes" if you want to deploy the QA
>>> test server')]
>>>
>>> }
>>>
>>> }
>>>
>>> }
>>>
>>> stage('deployQA') {
>>>
>>> when {
>>>
>>> environment name: 'DEPLOY_TO_QA_TEST', value: 'yes'
>>>
>>> }
>>>
>>> steps{
>>>
>>> /* deploy the build to a QA environment */  }
>>>
>>> }
>>>
>>>
>>> stage('Confirm deploy to UAT'){ // prompt user }
>>>
>>> stage {'deployUAT') { /* deploy the build to a PreProd/User Acceptance
>>> Testing enviornment */}
>>>
>>>
>>> stage('Confirm publish to Nexus'){ // prompt user }
>>>
>>> stage('publish') {
>>>
>>>    steps {
>>>
>>>      // mvn deploy -DskipTests (just deploy - don't rebuild)
>>>
>>>    }
>>>
>>> }
>>>
>>>
>>> Basically, I want to design my Jenkins pipeline to be my staging process.
>>> The problem I have is I'm not sure how I can only deploy at the end of
>>> the pipeline.  When maven runs the deploy lifecycle, it will run
>>> through all the other stages and reassemble my binaries, which
>>> technically are not longer the same as those that were approved.  So
>>> consequently, my binary hashes that were approved earlier in the
>>> pipeline are not the same hashes that are deployed in Nexus.
>>>
>>> I realize that i can probably do some work and use the Reproducible
>>> Build plugin (https://urldefense.proofpoint.com/v2/url?u=https-
>>> 3A__zlika.github.io_reproducible-2Dbuild-2Dmaven-2Dplugin_&d=DwIBaQ&c=
>>> RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=
>>> nSxyAsyxa1Izff8ULe7vW8u084madbci-hLPsiLpxeU&m=Og2S17jc02JRwm-oHae8UQmi
>>> Ig_
>>> ygbBRL0EQoB_Wvuw&s=zipmYyPLpmFYv1RsnquZtQMf0-HoYoix12SZj6gD2jM&e=),
>>> but that too comes with drawbacks (I want build timestamps in my
>>> Manifest files, and zip entries, etc).
>>>
>>> Am I sunk?  Is my only hope to wait until Sonatype releases Staging
>>> repos for Nexus 3.x sometime in Q2?  Or is there some other way I can
>>> work around this?
>>>
>>> How is everyone else handling this situation?
>>>
>>> Thanks for sharing.
>>>
>>> Eric
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Looking for recommendations how to best use Maven in a muti-stage Pipeline build to only deploy at the end

Robert Patrick
In reply to this post by Eric Benzacar
Downloading the artifacts (e.g., maven-dependency-plugin) and deploying them (e.g., maven-deploy-plugin) is the easy part, the tricky part is figuring out a "unique version number" of the artifacts to pass from one pipeline step to the next so that you are picking up the "right artifacts" from the staging repositories.  This is particularly important when multiple build pipelines are running in parallel...  

We have solved this in various ways in the past; for example, changing the version number from a SNAPSHOT version to a concrete version number based on a timestamp (e.g., 1.0.0-20180214-092053.123) prior to the running the automated build.  Unfortunately, none of the ways we have found so far were particular elegant so I it is hard to recommend a particular approach since they all have some drawbacks.

Maybe someone else has a better way to solve this problem...

-----Original Message-----
From: Eric Benzacar [mailto:[hidden email]]
Sent: Wednesday, February 14, 2018 9:16 PM
To: Maven Users List <[hidden email]>
Subject: Re: Looking for recommendations how to best use Maven in a muti-stage Pipeline build to only deploy at the end

Robert,

I think you misunderstood my problem.  It isn't a problem of defining different distributionManagement repos; like you said that is fairly easily handled using properties or -D parameters.

It is more the question of what kind of plugin is required to run which
says:
- download all the artifacts that this multi-module project previously built
- deploy all the artifacts that were just downloaded

I cannot think of how to design/configure a project to accomplish something like that.  I'm guessing I would have to create a profile which explicitly declares each artifact produced by the multi-module project in order to do a dependency:copy, and then similarly, use a build-helper:attach all the artifacts that were just downloaded.  But that seems like a painful solution.  Am hoping there is something easier?

Thanks,

Eric


On Wed, Feb 14, 2018 at 9:45 PM, Robert Patrick <[hidden email]>
wrote:

> Inline...
>
>
> -----Original Message-----
> From: Eric B [mailto:[hidden email]]
> Sent: Wednesday, February 14, 2018 8:27 PM
> To: Maven Users List <[hidden email]>
> Subject: Re: Looking for recommendations how to best use Maven in a
> muti-stage Pipeline build to only deploy at the end
>
> Hi Robert,
>
> Thanks for the suggestions.  To be honest, I never thought of using
> regular repositories as staging repos.  That being said, I do see a
> hiccup with the option that I'm not sure how to address.  In the case
> where a build is rejected during the pipelline stage, how do I delete
> it?  I haven't found a plugin that would allow me to delete a build from a repo.
> And with a multi-module project, there are many artifacts that would
> need to be rolled back.  I guess I could simply configure the repo to
> allow re-deployment of the same artifact to these "stage" repos, but
> if ever a deployment fails part way through, I end up with a repo in
> an unstable manner (ie: there is no way to ensure that all modules are
> promoted at the same time).
>
> [Robert] Typically, I would solve this problem by simply putting
> expiration dates on artifacts in the staging repositories.  I have
> never used Nexus but Artifactory allows you to limit artifacts in the
> repository by size, last time accessed, etc.  No need to keep
> artifacts in the "staging repositories" for more than say, one month,
> anyway since they were either good (and promoted all the way to the
> release repository) or bad (and are not interesting after they fail).
>
> All that being said, the last question would be how to retrieve and
> redeploy these artifacts from one repo to another?  Changing the
> distributionManagement settings is simple enough, but how would I
> instruct maven to download all the modules from a multi-module build
> and then deploy them?  I guess I could hack a profile in the parent
> pom where the different modules are defined with a
> dependency:copy-dependencies goal, but how do I ensure all artifacts
> that were just downloaded are then deployed to the next repo in the
> stage?  I would think the only way to do this would be writing my own
> custom plugin, unless you can think of some other mechanism to accomplish this?
>
> [Robert] Another way to do this is to use properties for the key
> values for the repository URLs in the <repositories> and
> <distributionManagement> sections of the POM.  You simply set a
> default value for the properties in the root POM (e.g., with
> appropriate values for the repositories used by the developers) and
> then override the property values with Maven command-line -Ds (or
> profiles) in the Jenkins build steps to handle pulling and promoting to the right places...
>
> Thanks,
>
> Eric
>
>
>
> On Wed, Feb 14, 2018 at 10:01 AM, Robert Patrick <
> [hidden email]>
> wrote:
>
> > While Nexus may not support "staging repositories", it certainly
> > supports more than one repository so why not just create one or more
> > repositories that serve as staging repositories.  For example,
> >
> > Pipeline Steps:
> > 1.) Trigger a build based on source check-in and push to stage1 repo
> > if build "succeeds"
> > 2.) Pull artifacts from stage1 repo, run stage 2 tests, and push to
> > stage2 repo if tests succeed.
> > 3.) Pull artifacts from stage2 repo, run stage 3 tests, and push to
> > stage3 repo if tests succeed.
> > 4.) Pull artifacts from stage3 repo, run UAT tests, and push to
> > release repo if tests succeed.
> >
> >
> >
> >
> > -----Original Message-----
> > From: Eric B [mailto:[hidden email]]
> > Sent: Wednesday, February 14, 2018 8:30 AM
> > To: Maven Users List <[hidden email]>
> > Subject: Looking for recommendations how to best use Maven in a
> > muti-stage Pipeline build to only deploy at the end
> >
> > I'm looking for recommendations for the best way to use Maven in a
> > multi-stage Jenkins pipeline build to deploy only at the end.  At
> > the moment, I'm using Sonatype Nexus 3.x, which means i don't have
> > the
> benefit
> > of staging repos.   Consequently, I have to ensure that the only released
> > versions of my libraries/application are final - they've passed QA.
> > Additionally, the team wants to ensure that the version numbers are
> > always incremental and every version in the repo is a consumable
> > version (ie: I cannot deploy a version 1.2.3 which has not passed
> > QA/Acceptance Tests, and furthermore, I cannot deploy a 1.2.2
> > followed
> by a 1.2.4).
> >
> > What that requirement translates to is that I have to ensure that
> > the binary built is fully tested before promoting it to Nexus. (and
> > that I shouldn't be appending build numbers to the maven version number).
> >
> > In my mind, I would like to do something the following in a Pipeline
> build:
> >
> > stage('build') { steps { sh 'mvn clean install'} }
> >
> > stage('Confirm deploy to QA'){
> >
> > steps {
> >
> > checkpoint 'test server deployed'
> >
> > script {
> >
> > env.DEPLOY_TO_QA_TEST = input message: 'User input required',
> >
> > submitter: 'authenticated',
> >
> > parameters: [choice(name: 'Deploy to acceptance test server', choices:
> > 'no\nyes', description: 'Choose "yes" if you want to deploy the QA
> > test server')]
> >
> > }
> >
> > }
> >
> > }
> >
> > stage('deployQA') {
> >
> > when {
> >
> > environment name: 'DEPLOY_TO_QA_TEST', value: 'yes'
> >
> > }
> >
> > steps{
> >
> > /* deploy the build to a QA environment */  }
> >
> > }
> >
> >
> > stage('Confirm deploy to UAT'){ // prompt user }
> >
> > stage {'deployUAT') { /* deploy the build to a PreProd/User
> > Acceptance Testing enviornment */}
> >
> >
> > stage('Confirm publish to Nexus'){ // prompt user }
> >
> > stage('publish') {
> >
> >     steps {
> >
> >       // mvn deploy -DskipTests (just deploy - don't rebuild)
> >
> >     }
> >
> > }
> >
> >
> > Basically, I want to design my Jenkins pipeline to be my staging process.
> > The problem I have is I'm not sure how I can only deploy at the end
> > of the pipeline.  When maven runs the deploy lifecycle, it will run
> > through all the other stages and reassemble my binaries, which
> > technically are not longer the same as those that were approved.  So
> > consequently, my binary hashes that were approved earlier in the
> > pipeline are not the same hashes that are deployed in Nexus.
> >
> > I realize that i can probably do some work and use the Reproducible
> > Build plugin (https://urldefense.proofpoint.com/v2/url?u=https-
> > 3A__zlika.github.io_reproducible-2Dbuild-2Dmaven-2Dplugin_&d=DwIBaQ&
> > c= RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=
> > nSxyAsyxa1Izff8ULe7vW8u084madbci-hLPsiLpxeU&m=Og2S17jc02JRwm-oHae8UQ
> > mi
> > Ig_
> > ygbBRL0EQoB_Wvuw&s=zipmYyPLpmFYv1RsnquZtQMf0-HoYoix12SZj6gD2jM&e=),
> > but that too comes with drawbacks (I want build timestamps in my
> > Manifest files, and zip entries, etc).
> >
> > Am I sunk?  Is my only hope to wait until Sonatype releases Staging
> > repos for Nexus 3.x sometime in Q2?  Or is there some other way I
> > can work around this?
> >
> > How is everyone else handling this situation?
> >
> > Thanks for sharing.
> >
> > Eric
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Looking for recommendations how to best use Maven in a muti-stagePipeline build to only deploy at the end

Brian Fox-2
In reply to this post by Eric B
On Wed, Feb 14, 2018 at 9:31 PM, Eric B <[hidden email]> wrote:
> Bernd,
>
> Nexus 3.x does not support staging repos b/c they are rewriting the entire
> platform to support not just Maven artifacts, but any type of repo-based
> artifact.  Ex: docker images, npm dependencies, etc...


This is true and it's getting close to being done. Also included is a
pipeline plugin for Jenkins to help you effect the promotions
directly.

>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]