Release Engineering as a Force Multiplier

In this post, I will post my understanding and its keynotes about http://oduinn.com/blog/2013/06/26/release-engineering-as-a-force-multiplier-at-google-techtalks/ , http://oduinn.com/blog/2013/06/07/release-engineering-as-a-force-multiplier-relengcon-2013/

Nowadays, Release Engineering is important to the success of every software project. People might find that more and more companies provide positions in Release Engineering and I am optimistic that there will be more and more people become professional Release Engineers in future. I am surprised that the world’s first ever Release Engineering conference was held in SF on this May 20th 2013! The industry should have realized the value of Release Engineering earlier!

Building an effetive pipelines (in my current company, we call it ‘flow’) requires a different mindset to writing a great software product. Release Engineers != Developers. I ever interviewed with some candidates which have good background in coding however they seems don’t have a good sense in Release Engineering. In their mindset, they just think that release engineering is just a job writing shell/perl scripts and nothing much! Hmmm, I understood that it is very common as Release Engineering is not taught as a discipline in any computer science. When I graduated from university in 2009 and started my career as a Release Engineer, I ever had any sense of Release Engineering. In my first 2-3 years, I had to learn on-the-job. This limits the effectiveness of Release Engineering to what can be learned on the job, and because information isn’t widely shared, its hard to learn from other people’s mistakes or successes. I am planing to write a booklet about Release Engineering to introduce the concept of Release Engineering, how it works, and what tools/scripts they will utilize, and how to be an seasoned release engineer, etc. That is also the reason why I opened this blog – I want to introduce Release Engineering to others. 🙂

When an organization encounter that its software delivery is slower and the product qualities are worse than before, and find that increasing headcount in dev & QA could not ease this pain. Yeah, show off time of Release Team. Release Engineers make entire organizations more effective while dev make good quality software. In the keynote, the speaker lists below data from his org (Mozilla):

Before & After

How quickly can we ship a chemspill release?

4-6 weeks 11 hours

How long to ship a “new feature” release?

12-18 months 6 weeks

How many active code lines?

1 1/2 42

How many checkins per day?

~15 per day 325 per day

I also have similar experience in my projects. We have nightly builds, checkin builds, branch builds, mainlines builds for different platforms and versions. Without this kind of build & release support, our dev & qa could not focus on their daily activities. Besides build & release, I also provide internal automated tools for them.

Who are Release Engineers’ customers? Haha, interesting topic. In the keynote, speaker provides two kinds of customers, Founder(s), CxO and VP Eng. Think those people are too big to us? Let me share you a real lesson. In my first job, I had ever done an info sharing session with one of our big boss who came from Headquarters by sharing her about Release Engineering team and how it works. However I had only 1.5 years working experience and I was still a newbie in Release Engineering. I had a bad presentation in this session because in this session, I shared her about what tools/ what techonoliges we used however seems she was no interested in that. In next paragraph, I will show what they are interested in. Here, I just want to convey that as a release engineer, you are closer to upper management than you think. Upper management they don’t care what tools/languages/technologies you are using, what they concerns are, how fast software can be delivered and how are the product quality. Release Team is in a good position to answer these kinds of concerns. Besides upper managements, we also have to walk around, chat with Dev+QA and find what do they care about and what do they need? And then do our best to comfortable them and improve their productivities. Basically we should be able to answer below problems (They are also the things what does upper management care about):

• fallout from shipping bad release
• fallout from slow response to security problem
• losing customers because of buggy software

missing release schedules during feature wars
(webM, …)
• turning away potential revenue
• infrastructure limits to hiring more developers

http://oduinn.com/blog/ is a very good source to learn how Release Engineering runs in a source organization.

 

Infrastructure as code

As a release engineer, I design/develop automation to serve core responsibilities when appropriate, and create/Enhance internal automation tools on demand for development/QA and IT teams. Like, automated tools to log bugs, parsing logs, fixing build issues, machines issues, build scripts, tools to maintain source codes, generating reports, etc. I love this job as it help improve the productive of the organization. I found a very interesting post ‘How Infrastructure Engineers are like Drug Pushers‘. In the post, the author said, an infrastructure engineers are just like Drug Pushers. 🙂

Here are their ‘evidences’.

Drug Pusher Infrastructure Engineer
The product can be addictive The tools can be addictive
Users rarely admit we exist Developers rarely admit we exist
Users don’t talk about us openly Developers don’t talk about us openly
The product make most users feel good The tools make most developers feel productive
Users go nuts without the product Developers go nuts without the tools
Users would like these supplies to be free Developers expect these tools to be free
When the product doesn’t work for them anymore, they expect new products that work exactly like the old products, but better When the tools don’t work anymore, new tools are expected, but they must work exactly like the old tools, but better
When the users are under the influence and happy, nothing else matters When the developers are happy and everything is working fine, nothing else matters

The tools can be addictive — In my current company, release team provides tools to manage the dual checkin of source codes. Yes, they love our tool. — They raised more than 30,000 requests in the past year! 🙂 How could we image if they or release team have to do these tedious jobs manually?

Developers rarely admit we exist — Bloody true. Release engineers are not coding for customers (We are coding for developers). Okie, so many people might think, “Release enigneers are not generating profit.” — Too bad. 😦

The tools make most developers feel productive — ASA developers check in codes into repository, it will trigger a checkin build and it will provide feedback quickly. That makes our developers be more confident in their codes and improve their productive. Our tools also help them just focus on programming.

Developers go nuts without the tools — Hahaah, can anyone image that, without the support of release engineers, dev could focous on their programming?

Seriously, let’s come back to the topic ‘Infrastructure as code‘.

In post Concise Introduction to Infrastructure as Code, it has a well defined of what is Infrastructure as Code,

The end goal of infrastructure as code is to perform as many infrastructure tasks as possible programmatically. Key word is “automation.”

Yes, Automation Automation and Automation. Automate Everything if those steps are reproducible. Server configuration, packages installed, dependencies managements, automated testing, reports, build artfiacts, deployments, bugs managements, etc. All of them should be automated and remove all of the manual steps prone to errors. Computers won’t do things wrong unless engineers provides them wrong directives.

I believe that every build team has a lots of internal automated tools work separately. I think we can move further and integrate them into a basic infrastructure platform and unify the data input/outputs (generally, the data is, change set/list, bugs, and build artfacts). We should see infrastructure as a single target and should not treat them separately, like IT, DevOps, QA, Build, Release, and Supports. Ideally, the data flow should be,

a new bug triggered automatically-> notify QA/DEV -> Verified by QA/DEV-> provides environment to QA/DEV to debug-> checkin code-> build the new code-> code quality -> dependencies and changes reports-> deploy new code-> notify QA/DEV the result-> passed -> includes into the build cycle.

In above steps, they should be automated as much as possible.

Yes, it is easier said than done! But as we have already known the direction, ASA we decide to go, we can achieve the goal ‘automated everything’ one day with our hard working!

Continuous Database Integration

Database integration is one of the toughest parts in Integration. In this post I will review some keynotes in DI.

Firstly, all of the files related to DI should be source controlled. Ensure that your database scripts have been tested and verified.

Secondly,  let’s list all of the reproducible database steps.

1. Delete a database.

Delete a database and its data so later you can create a new database with the same name.

2. Create database.

Use DDL files to create new database.

3. Import data.

Use insert/import/etc. scripts to import data.

4. Migrate database and its data.

Migrate database schema and its data to a new environment.

5. Modify database objects.

Use DDL files to modify database objects.

6. Update testing data.

7. Backup/restore data.

My experience is,

For every build, I need to run the step 1, 2, 3 in an automated flows (basically this step will take 2-3 days), and then in step 4, I will use Oracle VM system to templatize my base environment as a template, so DEV/Testers can just clone their environments from the template and then they can have the same data with the base environment in minutes.

For step 5,6 we use Oracle Database Edition Technology. Every time we need to apply DDL and DML, we will apply them in the PATCH edition and only after that is successful we will cut over it to RUN edition.

For step 7, we just simply use oracle imp/exp tool to implement that.

During database development, DBA should not involve into data migration because this part should be strictly handled by automated flows and DBA should take part in Database performance and other priorities. And by providing DEV a template, DEV can have a clone environment in mnutes and we call those environments as Sandbox in that way DEV can develop independently.

Continuous-Delivery-Maturity-Model

For a practitioner of Release Engineering, Continuous Delivery is not merely just utilizing tools and doing some automated activities. It has to request everyone involved in the process of delivery to coordinate together to achieve the delivery goal. In this article I will introduce two Continuous-Delivery-Maturity-Models and explore how we can practice them in our continuous delivery activities.

The first Continuous-Delivery-Maturity-Model I want to introduce here is from http://www.infoq.com/articles/Continuous-Delivery-Maturity-Model

The model defines five maturity levels: base, beginner, intermediate, advanced and expert in five categories Information & Reporting, Test & Verification, Build & Deploy, Design & Architecture, and Culture & Organization. I think the advantage of this model is, it considers not only from the view of DevOps, it also consider Culure & Organization. Sometimes, the organization and it’s culture are probably the most important aspects to consider when aiming to create a sustainable Continuous Delivery environment. I ever worked for a company with a specified Build & Release Team however due to the company culture the team get dismissed after some years. So understanding how maturity of your company is can give you an understanding how far the Continuous Delivery and even the Release Engineering team can go.

However from the view of Release Engineers, the maturity of Culture & Organization is a bit out of control. A Release Engineer sometimes can not do much to change the Culture & Organization. And, in this model, it also introduces the maturity of Design & Architecture, we all know that as Release Engineer we need to involve into the architecture Design and also understand that the design and architecture of products will have an essential impact on our ability to adopt continuous delivery. But now, what I just want to share in this post is, how to manage your continuous delivery given that you have to go with your current architecture and organization culture. Hereby I will introduce another Continuous-Delivery-Maturity-Model below. It is a model listed in Book <Continuous Delivery – Reliable Software Releases through Build, Test, and Deployment Automation> and Continuous Delivery: A Maturity Assessment Model.

The model has 5 levels in categories ‘Build Management & CI’, ‘Environment & Deploy’, ‘Release Management (Compilance)’, ‘Testing’, ‘Data Management’, and ‘Configuration Management’. Let me summary them as below:

Level-5: Optimizing,

Build Management & CI: Every team has a fixed time to update each other, discuss integration problems, and can fix problems in an automated, faster feedback, and visual way.

Environment & Deploy: It can manage all of the environments effectively, all of the environment preparation can be automated and if possible, can use virtualization technology.

Release Management (Compilance): DevOps and delivery teams have a fixed time to co-work together to manage delivery risk and shorten the SDLC.

Testing: The deployment on Production works in most of the time and no rollback. All of the bugs can be found and fixed in a short time.

Data Management: The data for database performance testing and deployment can be reused and can provide feedback between two release cycles.

Configuration Management: Can review regularly whether configuration management can support an effective cooperation among teams, a fast development and an audit-able changes management.

Level-4: Quantitatively,

Build Management & CI: The build can not be in a Fail status for a long time and it should collect build metrics.

Environment & Deploy: The deployment management is being tracked carefully and the process of release and rollback can be tested.

Release Management (Compilance): The health of environments and application can be monitored in an early bird and cyclic way.

Testing: The quality can be tracked and metrics-ed and all of the non-functional requirements can be defined and metric-ed.

Data Management: It should monitor and optimize the database and the database upgrade and rollback testing can be taken care for every deployment.

Configuration Management: Developers should commit once into the mainline every day and should create a branch only when it is time to have a release.

Level-3: Consistent,

Build Management & CI: Check-in build and check-in testing. Dependencies can be well managed. Tools and scripts can be reused.

Environment & Deploy: Software Delivery can be automated and it should use an unified to deploy software to all of the environments.

Release Management (Compilance): Define and conduct change management. All of the policies and compliance can be satisfied.

Testing: Automated unit testing and acceptance testing. Testing is part of development.

Data Management: Changes in database can be automated and be part of the deployment pipeline.

Configuration Management: Libraries and dependencies can be well managed.

Level-2: Reproducible,

Build Management & CI: Can build and test software automatically and regularly. All of the builds should based on the latest code from code repository.

Environment & Deploy: All of the environment configuration should be sourced control and separated with source code. It will not cost too much time and effort to create a new environment.

Release Management (Compilance): It is a bit painful to release software but the release should be in a good quality. Part of the configuration from requirement inception to delivery can be tracked.

Testing: Automated testing is part of the development of user stories.

Data Management: Changes in database can be automated and can be sourced control.

Configuration Management: Source codes, configuration, build & deploy scripts, and data migration can be sourced control.

Level-1: Retardative,

Build Management & CI: Software build is in a manual way and can not manage artifact and reports well.

Environment & Deploy: Software deployment is in a manual way and the software packaging is environments dependent. The preparation of environment is manual.

Release Management (Compilance): Can not release in a good quality frequently.

Testing: Can do manual testing only after developments finish.

Data Management: Database migration is in a manual way and doesn’t do version control.

Configuration Management: No source control and the commit is not frequent.

This model is a very good and with a high practicability. As a release engineer, we have to involve into all of the categories ‘Build Management & CI’, ‘Environment & Deploy’, ‘Release Management (Compilance)’, ‘Testing’, ‘Data Management’, and ‘Configuration Management’ in our daily jobs. I think this model can help enhance our Continuous Delivery definitely.

I believe that we can use this model as our guide and it will help us to do continuous improvement.

Problems in Software Delivery

As a software engineer, the most important thing we need to resolve is, how to deliver the product to its customers?

In this page, I will share my thought in this.

Let me introduce a very typical continuous integration pipeline here. You can refer to my earlier post Continuous Integration for more details.

Build=>Unit Testing => Code Analysis => Staging => Deployment => Delivery

For many companies, the Release Day is the busiest day and why? That is because for most of the projects the risk in delivery is highest!

Let’s consider a scenario. In Release Day, Operations Team needs to prepare the OS environment, and then install 3rd-parties libraries and software. After that, Operations Team needs to transfer the to be deployed applications to the server and then follow the deployment guide provided by development team to configure the system. And if it is being deployed to a distributed environments, Operations Teams has to do above steps node by node. Okie, now everything is almost done. Operations Team tries to startup the applications however it fails out! — I think many engineers would have the similar experience as here.

Below is the summary of problems in software delivery:

1. You deploy your software manually – even for your testing, staging, demo environments.

The characteristics of this practice are:

# You have a very detailed document and you have to follow the document strictly to deploy your application

# You need to do manual testing to verify whether your deployment works.

# You need to work with development team very frequently to fix deployment issues.

# If your deployment is distributed, then you have to deploy them node by node.

# Deployment will take some time.

# The result of deployment is unknown. You cannot know whether your deployment can work or not.

# Your team doesn’t deploy or test on staging environment frequently. – Here I defined ‘Staging Environment’ as an environment with the same configuration as Production.

# You need to configure your production environment manually. In other word, you can not reproduce a duplicated production environment for other purposes in minutes.

In a word, you can not deploy your applications by clicking an ‘Enter’ button.

Okie, we all know that we can use Automated Continuous Integration/Delivery to fix above problems in Software Delivery. Fast delivery is very important nowadays, we need to have a fast, effective, and reliable way to delivery high quality and value products.

To achieve this goal, we need to delivery the products frequently and automatically:

1. Automated. If we can not automate our build, deploy, testing and delivery, then it is not reproducible. The result of every deployment is unpredictable because software configuration, system configuration, and the procedures can not be reproducible. That means we can not provide good quality products through our release engineering.

2. Do it frequently. If we can deploy very frequently then the difference between deployment with last deployment will be minor and this will reduce the risk dynamically and also it is easy to roll back the deployment if any issue happen.

Configuration Management

Configuration Management – In this post, I will just conclude some principles in CM.

Introduction

Configuration management (CM) is a process for establishing and maintaining consistency of a product’s performance, functional and physical attributes with its requirements, design and operational information throughout its life. ” 

Below is the checklist you can use to check whether you have adopted the CM in your team or not and you should “YES” to all the question:

1. Can you reproduce any of your environments – Build, Test, Staging, and Deploy, etc. including OS versions, patchings, networking configuration, technologies stacks, and other software applications and setting deployed upon your environmnets?

2. Can you easily deploy your new changes into your environments in an incremental manner?

3. All your changes in configuration could be tracked?

4. All of the CM rules/policies can be conducted strictly?

5. Is it that anyone from development team, testing team, PMs, etc. can easily visit these CM information?

6. Will your CM strategies delay the software delivery?

Version Control

One of the essential elements of CM is Reproducible. So deploying a Version Control System in your organization is a MUST. There are lots of Version Conrol Tools. Like RCS (Revision Control SYSTEM) shipped with Linux, CVS (Concurrent Versions Control), SVN (Subversion), and Perforce, etc. In this article I will not compare their advantages or disadvantages. But my recommend is, if your team wants to use commercial product, then Perforce will be your choice. If your team wants to use opensource products, I strongly recommend SVN than CVS.

Version control all you have

Here the term ‘Version Control’ is not the same as ‘Source Control’ as Version Control is not merely for Source Control only. In my opinion, all of the artifacts related to your developments should be under version control.  Development codes, testing codes, database scripts, build/deploy scripts, documentations, libraries files, and all of the configuration files, even compilers and tools should be under version control. In this way, it will help new team memmbers join your team easily.

Commit code frequently into mainline

When adopt Version control, there are two principles. Firstly, you should commit codes very frequently so you can find out the build errors supposed you have check-in builds and then you can roll back to the last working version. Secondly, your team members can see what you changed and they can work based on your latest changes.

In some of the development teams, they will create branch to do their developments however that will cause below problems,

1. It is breaking the Continuous Integration principles. As in CI, it requires that we should integrate ASAP and working with branch will delay the integration.

2. It is difficult to merge those branches together if they go too far.

3. It is difficult to do code refactoring.

So, we should commit code frequently and commit them into mainline.

Have meaning comments when commits code

All of your commit should give them a making sense comments. Like, “This commit is for bug#123456, and it would introduce method getId();”. It is humanity to give your commit a meaning comment.

Dependencies Management

In a SDLC, the most common dependencies are 3rd-parties libraries and the dependencies introduced by those 3rd-parties.

Libraries management

Those libraries usually exist with binary files, like a junit.jar, unless you are using interpreting languages. Actually there is argument that whether we should have those libraries under version control. My suggestion is that we should also have version control for those libraries. Yes it is true that you can download them from internet if you want to deploy a new environment and those libraries basically will not get changed. But let’s consider scenario: Your project is using a 3rd-party library ABC-3.2.1.jar from http://repository.abc.com however due to an update in  ABC-3.2.1.jar (Version 2), then your configuration will get updated automatically and it will break your build or introduce potential problems. Maven provides a powerful in dependencies management and it will help you save a copy locally as local repository if you configure it works like that.

Modules management

Modules mean your internal projects. Supposed you are using Maven as the Build Management tool, then all of your internal projects can be deployed into the local repository as a module.

Software Configuration Management

As a critical component, configuration information work together with product code and its data to build the applications. We can control the build, deployment, and live of a software product via configuration information. A delivery team should consider seriously how to configure those information and how to manage them during the full of SDLC.

Configuration VS. Flexibility

Everyone wants to have the highest flexibility but sometimes we just can not have all of the good things in the same time. We need to sacrifice part of the flexibility for a better product quality. The risk of changing in configuration management is lower than changing codes. Supposed that development has its highest flexibility then all of the configuraiton information will be stored in the source codes and any changes will cause a change in source code. That is too much.

Configuration Categories

We can configure system during its processes of build, deployment, testing, and publish. Below is the categories list of configuration information:

1. Configuration information for building software into binary.

2. Configuration information for packaging into a war, ear, etc.

3. Configuration information for deploying software.

4. Configuration information for starting up/running the software.

Principles of Configuration management

We should treat configuration information the same as source code, manage them properly and test them. Below are some points we need to consider,

1. When and what configuration setting we can introduce during a SDLC? For example, for the setting of junit.jar, we should introduce them only in testing phase; for JDBC pool setting, we should have different setting during different phases (like System Testing and Production).

2. All of the configuration information should be stored in the same repository as source code but the detailed information should be stored in a different place. User passwords, web console passwords should not be sourced control for safety.

3. The instantiation of a configuration file should be automated.

4. There should be a naming convention for all of the configuration settings.

5. DRY (Don’t Repeat Yourself). Just ensure that every configuration item works for a special purpose and there are no duplicated item with the same purpose.

Environment Management

We can utilize Virutalization Technologies to do environment management. In my current company, we are using Oracle VM to templatize any environments and it can provide a duplicated environment just in a click. All of the changes in the environment can be tracked.

Summary

Configuration is the basic requirement to do Continuous Integration, delivery management. We should version control of everything required to build the software. Of course, we need to balance between Flexible and Configuration (I will have a post later to introduce Maven’s Convention over Configuration.).

Patches Classification

“A patch is a piece of software designed to fix problems[1] with, or update a computer program or its supporting data. This includes fixing security vulnerabilities[1] and other bugs, and improving the usability or performance. Though meant to fix problems, poorly designed patches can sometimes introduce new problems (see software regressions). In some special cases updates may knowingly break the functionality, for instance, by removing components for which the update provider is no longer licensed or disabling a device.” — from wiki

In Release Engineering, it has one methodology called Patches Management.One of the important thing is, how to classify those patches. For example, customer A may want to have its documents updated, customer B is expediting a very critical fix for security, and other customers are pursuing functional issues, or even customized fix. A well-defined Patches classification will definitely help improve the software quality.

Below are some patches terminology used by Oracle Company.

Critical Patch Update (CPU) now refers to the overall release of security fixes each quarter rather than the cumulative database security patch for the quarter. CPU are collection of security fixes for Oracle products and they are available to customers with valid support contracts. So here my suggestion is, in a software company, it should provide a CPU patch to its customers in a scheduled date. Also it is a good chance to extend its support contract and make money from this kind of proactive support. You can refer to Oracle CPU for more details.

Patch Set Updates (PSU) are the same cumulative patches that include both the security fixes and priority fixes. The key with PSUs is they are minor version upgrades (e.g., 11.2.0.1.1 to 11.2.0.1.2). Oracle PSU is a new patching strategy whereby the DBA can choose only ‘recommended’ and ‘proactive’ patches. PSU is a sub patch of CPU. CPU will contain both the PSU and CPU, so the DBA may choose to apply just the CPU or apply apply all patches in the PSU patch bundle (which includes additional fixes). It is more flexible.

One-off patch, it is for one off use only and if it is not urgent, customers should wait for the CPU or PSU, however if it’s urgent then customers can choose to apply a One-off patch. All of the one-off patches should get included into PSU or a majoy upgrade patches.

Bundle Patches, it is a patch with some one-offs. It can help save time to apply patches. Supposed you want to upgrade a product and it requires you to apply 10 one off DB patches, then you can just apply the bundle patch that contain those 10 one offs.

Diagnostic patches, it is intended to help diagnose or verify a fix or a collection of bug fixes. For example, if developers want to debug in customers’ site, then release engineer could release a diagnostic patch.

Interim patches/Interim patches for security bug fixes, it contain a single bug fix or a collection of bug fixes for customer-specific security bug fixes or provided as required.

You may also define a patch type as ‘Document patches‘.

Below is the information from http://docs.oracle.com/cd/E25290_01/doc.60/e25224/patchesupdates.htm

The Patches and Updates tab enables you to view and download recommended patches and updates for your Oracle products. You can download the following patch types:

  • Interim patches – contain a single bug fix or a collection of bug fixes provided as required

  • Interim patches for security bug fixes – contain customer-specific security bug fixes

  • Diagnostic patches – intended to help diagnose or verify a fix or a collection of bug fixes

  • Bundle Patch Updates (BPUs) – a cumulative collection of fixes for a specific product or component

  • Patch Set Updates (PSUs) – a cumulative collection of high impact, low risk, and proven fixes for a specific product or component

  • Security Patch Updates – a cumulative collection of security bug fixes

  • Sun hardware and firmware

Oracle recommends that users apply patchsets instead of individual patches. Because of the very large number of individual patches, limiting the search for patchsets provides faster results.

Conclusion:

A well-defined patches clarification will definitely help improve the quality of customers support. And it will help development have a better control of our patches quality and project management. Development team and testing team can have a better justification of patches expedition.

Continuous Integration – Session

“Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.” – Martin Fowler.

In some companies which don’t have a Release Team they don’t adopt CI even thought they claim that they are going with Agile Development. In my opinion, it is a fake Agile if no CI in their SDLC.

Nowadays it is very common that you have to get ready to deploy your artifact in anytime to satisfy the business requirement and the era that without a professional release engineer that you still can produce good quality product in a timely manner has gone! More and more organizations have their Build/Rlease Team, in other term, DevOps.

I ever had a sharing session with new hires. The session is about what is CI, and how we conduct CI, and the function of Build & Release Team. I hope that through that session ppl especially those new hires could understand well our SDLC.

I shared my slides below with necessary modification and pages removed to satisfy the company policy.

Below is a brief of my session,

WHAT IS CONTINUOUS INTEGRATION(CI)?

Maintain a Single Source Repository
Checkin & Automate the Build
Everyone can see what’s happening
Make it Easy for Anyone to Get the Latest Execute
Make Build Self-Testing
Automate Deployment

The value of CI:

Reduced risks.
Reduce repetitive manual processes
Generate deployable software at any time and at any place
Enable better project visibility
Establish greater confidence in the software product from the development team

CI is not used to find out compile error, although it can.

Compile is the most basic thing CI do, compile error is not acceptable
The target of CI is helping find out integration /deployment problems as earlier as possible.
Ideally, a successful build in CI should:
 1. Compile succeeded
 2. Passed all unit test
 3. Unit test coverage reach the acceptable rate
 4. Passed all functional test and regression test
 5. Passed performance test
 5. Passed user acceptable test if necessary
Any successful build of CI could generate a deliverable package, so CI could & should give confidence to team members that our product can be deployed to production at any time.

CI is one of core practices of Agile, effective CI need the whole team follow other practices, on the other hand, CI could work with other practices to make the whole project better.

Test Driven Development
Automation Testing
Coding standard adherence
Refactoring
Small releases
Collective ownership

BEST PRACTICES OF USING CI – DEVELOPER

Commit code frequently
Don’t commit broken code
Fix broken builds immediately
Write automated developer tests
All tests and inspections must pass
Run private builds
Avoid getting broken code

INTRODUCING BUILD RELEASE TEAM

Source code management –>
Source control system (like CVS, SCCS, Subversion) setup and maintenance
Setup and monitor daily continuous/production builds
Co-ordinate with the development team in case of build failures
Update build tools as per changes in the product output/strategies
Create branches and setup separate build system after a milestone releases
Create build storage archives for back tracking builds

Cross team co-ordination –>
Gather build specifications from the technical team
Document build instructions for the development team
Participate in release/milestone planning and scheduling with the product team
Announce and communicate vetted builds to QA and the documentation team
Work with the localization team to produce multi-language bits
Work with the sustaining team during product patches/upgrades
Coordinate with other dependent groups for their product milestone bits, e.g. aplication server, jdk, etc.

Build output –>
Setup kit server for build storage
Setup a cache server for storing other product bits or third party bits
Upload bits for internal and external customers
Create CD/DVD iso images and physical media
Code Quality Control–>
Setup code standard?
Monitor code quality trend

Software Engineering –>
Agile & CI governed
Automated the more the better
Workflow optimized

You can download the slide from Continuous Integration-Session

Career Path – Release Engineering

Release Engineering (“RE”) is a sub-discipline in Software Engineering concerned with the compilation, assembly, and delivery of source code into finished products of other software components. It is Release Engineering that making Software Engineering as manufacturing is to an industrial process. We can say that without RE, the Software Industry was still in primitive society.

Let me take Oracle EBS as an example,

How many product family (every product family will consist with around 10 small product) Oracle EBS have?

Oracle CRM
Oracle Financials
Oracle HRMS
Oracle Mobile Supply Chain Applications
Oracle Order Management
Oracle Procurement
Oracle Project Portfolio Management
Oracle Quotes
Oracle Transportation Management
Oracle Warehouse Management Systems
Oracle Inventory.
Oracle Enterprise Asset Management

And in the techology stacks, it has to support Linux-32/64, Solaris-32/64, AIX, HPUX-IA64, Windows; Weblogic, Oracle iAS, Oracle Forms, Oracle OC4J, Jrockit, SUN JDK, and comprehensive support plans for different customers, different release and different upgrade paths.

You can have a good understanding how complicated in EBS from,

http://www.oracle.com/technetwork/apps-tech/2011-june-ebs-roadmap-405789.pdf

https://blogs.oracle.com/stevenChan/

As software development evolves, it is more difficult and even impossible that without a professional release engineer in a software development team and they can succeed to provide good quality products in time. Actually I am a passionate practitioner of Book <Continuous Integration: Improving Software Quality and Reducing Risk> and book <Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation>. I strongly recommend you to read these two books if you are interested in RE.

An Release Engineer’s daily job,

1. Code Management,

It has lots of good code management tools, like CVS, SVN, Perforce, even with Linux’s RCS. Here has to work as an SCM.

2. Mainline/Branch management,

Manage mainlines, branches, branching strategies and the bugs attached to them. Here was like a product manager 🙂

3. Install/Upgrade Testings,

Install testing, fixes, bugs, and verify all of the possible upgrade paths for customers. Here works as a system testing engineer.

4. Report/analysis/triage any build compile issues,

Here works as a debuger. Find out who do the evil and expedite fixes. There are some nice build tools like Maven, Nant, ant.

5. Code qualities,

Leverage JUnit, Findbugs, PMD, Checkstyle, Sonar to enhance code quality. And it is easy to plugin them into a workflow build engine like Hudson, Cruise. Here works as a Quality Controller.

6. Integration issues,

How to integrate a product suite with a complicated tech stacks is a big challenge nowadays.

7. Virtualization,

Utilize VM tools to templatize the latest artifacts into templates. Works as IT guys.

8. Internal tools enhancement,

Release Engineers have to get ready to product tools to enhance productivity. Works as developers.

9. Database management/OS management

We can see that as Release Engineer, ppl have to involve into OS, Database, Middle tier, Languages, Build/Release, Testing, Code quality, IT, even requirement management!

So in my opinion, how to become a professional Release Engineer?

1. Firstly, you need to love the Software Industry and take it as your interesting but not merely a job!

2. Release Engineer is a Dev + Ops. (DevOps) + Test + IT. You need to develop your skills as a generalist.

3. Enhance some key skills, like Database, Mid-tier services, development skills. (That is why I took Oracle Database, Java certified).

4. Detailed in analysis, and always ready to help ppl. You have to have development sense and get ready to work after office hours sometimes as the build will fail in the midnight and get it fixed before testers yell to you in next morning.

5. Think from customers’ view sometimes. You are the last chance to find out issues before rolling out the products.

6. Be responsibility. When a build issue, exception, bugs occur, do NOT throw it to Dev/Tester right away although you can do like that. You should have a detailed and quick analysis that firstly.

Career path:

Release Engineer -> Have sense in Release Engineering. Good knowledge in tools/Scripts/tech stacks.

Sr Release Engineer -> Can do detailed analysis and deep knowledge and skills. Conduct Release plan.

Release Lead/Manager -> Compose Release plan and lead the team.