Building Microservices with Go Build seamless, efficient, and robust microservices with Go Nic Jackson BIRMINGHAM - MUMBAI < html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC- html40/loose.dtd"> Building Microservices with Go Copyright © 2017 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. First published: July 2017 Production reference: 1250717 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78646-866-6 www.packtpub.com Credits Author Nic Jackson Copy Editor Karuna Narayan Reviewers Magnus Larsson Erik Lupander Project Coordinator Vaidehi Sawant Commissioning Editor Kunal Parikh Proofreader Safis Editing Acquisition Editor Karan Sadawana Indexer Francy Puthiry Content Development Editor Zeeyan Pinheiro Graphics Jason Monteiro Technical Editor Vivek Pala Production Coordinator Nilesh Mohite About the Author Nic Jackson is a developer advocate working for HashiCorp.com; he has over 20 years, experience in software development and leading software development teams. A huge fan of mobile application and microservice architecture, he is constantly looking out for the most efficient way to reuse code and improve development flow. In his spare time, Nic coaches and mentors at Coder Dojo teaches at Women Who Go and GoBridge, speaks and evangelizes good coding practice, process, and technique. About the Reviewers Magnus Larsson has been in the IT business since 1986. He is an experienced architect in areas such as distributed systems, systems integration, and SOA. He is currently engaged in exploring the benefits of modern technologies such as microservices, container technologies, reactive frameworks, and mobile solutions. Magnus has a special dedication to the open source community for Java and has been active in various projects over the years. He enjoys exploring other languages and currently finds the Go language very interesting for building microservices. He is also a frequent speaker at conferences such as Cadec, Spring I/O, Jfokus, and jDays. Magnus lives outside Gothenburg, Sweden, with his family. When time permits, he enjoys cross-country skiing, which must be done either on roller skis or indoors, in the Gothenburg area for most of the year. He has worked for large corporations such as Digital Equipment Corporation, AstraZeneca, and Ericsson Hewlett Packard Telecom over the years. In 2000, Magnus co-founded Callista Enterprise AB, a Swedish-based consultancy company specialized in software architecture. Erik Lupander is a software architect and developer with over 15 years of professional experience. A lifelong computer and software enthusiast, he wrote his first GW-BASIC programs at the age of 7 back in the mid-80s. Erik holds an M.Sc. in applied informatics from the University of Gothenburg and has worked in a variety of roles in the software industry ever since. While JVM-based languages and architecture has been his bread and butter, he is a polyglot software craftsman at heart who, among other technologies, has embraced the Go programming language and microservice architecture. Erik has spoken at software conferences such as Scandev (2012), dev:mobile (2014), and Cadec (2016, 2017) about topics ranging from OpenGL ES to Golang and microservices. He lives just outside Gothenburg, Sweden, with his wife and two children. Aside from family, computers, and software, he enjoys alpine skiing, golf, and running, and he is an avid supporter of IFK Gothenburg. Erik is currently employed by Callista Enterprise AB, a Swedish-based consultancy specialized in software architecture. His previous employers include Siemens Medical Solutions, Epsilon IT, University of Gothenburg, and Squeed AB. www.PacktPub.com For support files and downloads related to your book, please visit www.PacktPub.com Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at service@packtpub.com for more details. At www.PacktPub.com , you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks. https://www.packtpub.com/mapt Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career. Why subscribe? Fully searchable across every book published by Packt Copy and paste, print, and bookmark content On demand and accessible via a web browser Customer Feedback Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at https://www.amazon.com/dp/ 1786468662 If you'd like to join our team of regular reviewers, you can e-mail us at customerreviews@packtpub.com We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products! Table of Contents Preface What this book covers What you need for this book Who this book is for Conventions Reader feedback Customer support Downloading the example code Errata Piracy Questions 1 Introduction to Microservices Building a simple web server with net/http Reading and writing JSON Marshalling Go structs to JSON Unmarshalling JSON to Go structs Routing in net/http Paths Convenience handlers FileServer NotFoundHandler RedirectHandler StripPrefix TimeoutHandler Static file handler Creating handlers Context Background WithCancel WithDeadline WithTimeout WithValue Using contexts RPC in the Go standard library Simple RPC example RPC over HTTP JSON-RPC over HTTP Summary 2 Designing a Great API RESTful APIs URIs URI format URI path design for REST services Collections Documents Controller Store CRUD function names HTTP verbs GET POST PUT PATCH DELETE HEAD OPTIONS URI query design Response codes 2xx Success 200 OK 201 Created 204 No Content 3xx Redirection 301 Moved Permanently 304 Not Modified 4xx Client Error 400 Bad Request 401 Unauthorized 403 Forbidden 404 Not Found 405 Method Not Allowed 408 Request Timeout 5xx Server Error 500 Internal Server Error 503 Service Unavailable HTTP headers Standard request headers Authorization - string Date Accept - content type Accept-Encoding - gzip, deflate Standard response headers Returning errors Accessing APIs from JavaScript JSONP CORS RPC APIs RPC API design RPC message frameworks Gob Thrift Protocol Buffers JSON-RPC Filtering Versioning APIs Semantic versioning Versioning formats for REST APIs Versioning formats for RPC APIs Naming for RPC Object type standardization Dates Durations Intervals Documenting APIs REST based-based APIs Swagger API Blueprint RAML RPC based-based APIs Summary 3 Introducing Docker Introducing Containers with Docker Installing Docker Running our first container Docker volumes Union filesystem Mounting volumes Docker ports Removing a container starting with an explicit name Docker networking Bridge networking Host networking No network Overlay network Custom network drivers Weaveworks Project Calico Creating custom bridge networks Creating a bridge network Connecting containers to a custom network Writing Dockerfiles Building application code for Docker FROM MAINTAINER EXPOSE COPY ENTRYPOINT CMD Good practice for creating Dockerfiles Building images from Dockerfiles Docker build context Docker Ignore files Running Daemons in containers Docker Compose Installing Docker Compose on Linux Service startup Specifying the location of a compose file Specifying a project name Summary 4 Testing The testing pyramid Outside-in development Unit tests httptest.NewRequest httptest.NewRecorder Dependency injection and mocking Code coverage Behavioral Driven Development Testing with Docker Compose Benchmarking and profiling Benchmarks Profiling Summary 5 Common Patterns Design for failure Patterns Event processing Event processing with at least once delivery Handling Errors Dead Letter Queue Idempotent transactions and message order Atomic transactions Timeouts Back off Circuit breaking Health checks Throttling Service discovery Server-side service discovery Client-side service discovery Load balancing Caching Premature optimization Stale cache in times of database or downstream service failure Summary 6 Microservice Frameworks What makes a good microservice framework? Micro Setup Code generation Tooling (CI/CD, cross platform) Maintainable Format (REST/RPC) Patterns Language independence Ability to interface with other frameworks Efficiency Quality Open source Security Support Extensibility What we learned about Micro Kite Setup Code generation Tooling Maintainable Format Patterns Language independence Efficiency Quality Open source Security Support Extensibility Summing up Kite gRPC Setup Code generation Tooling Maintainable Format Patterns Language independence Efficiency Quality Open source Security Support Extensibility A few lines about gRPC Summary 7 Logging and Monitoring Logging best practices Metrics Types of data best represented by metrics Naming conventions Storage and querying Software as a service Self-hosted Grafana Logging Distributed tracing with Correlation IDs Elasticsearch, Logstash, and Kibana (ELK) Kibana Exceptions Panic and recover Panic Recover Summary 8 Security Encryption and signing Symmetric-key encryption Public-key cryptography Digital signatures X.509 digital certificates TLS/SSL External security Layer 2 or 3 firewalls Web application firewall API Gateway DDoS protection Types of DDoS attack UDP fragment attack UDP flood DNS NTP CHARGEN SYN flood SSDP ACK Application security Prevention Detection Response Recovery Confused deputy How an attacker could bypass the firewall Scenario Attack Input validation Fuzzing TLS Generating private keys Generating X.509 certificates Securing data at rest Physical machine access OWASP Never storing session tokens in a URL Cross-site scripting (XSS) and cross-site request forgery (CRSF) Insecure direct object references Authentication and authorization Password hashing Adding a little seasoning Dictionary attacks Adding a pepper bcrypt Benchmarks JWTs Format of a JWT Secure messages Shared secrets Asymmetric encryption with large messages Maintenance Patching containers Software updates Patching application code Logging Summary 9 Event-Driven Architecture Differences between synchronous and asynchronous processing Synchronous processing Asynchronous processing Types of asynchronous messages Pull/queue messaging Push messaging Command Query Responsibility Segregation (CQRS) Domain-Driven Design What is DDD? Technical debt Anatomy of DDD Strategic design Tactical design Ubiquitous language Bounded Contexts Context Mapping Software Kafka NATS.io AWS SNS/SQS Google Cloud Pub/Sub Summary 10 Continuous Delivery What is Continuous Delivery? Manual deployment The benefits of continuous delivery Aspects of continuous delivery Reproducibility and consistency Artifact storage Automation of tests Automation of integration tests Infrastructure as code Security scanning Static code analysis Smoke tests End-to-end tests Monitoring Continuous delivery process Overview What is container orchestration? Options for container orchestration What is immutable infrastructure? Terraform Providers Terraform config entry point VPC module Output variables Creating the infrastructure Example application Continuous delivery workflow Build Testing Benchmarking Static code analysis Integration tests Deployment Smoke tests Monitoring/alerting Complete workflow Summary Preface Microservice architecture is sweeping the world as the de facto pattern for building web-based applications. Golang is a language particularly well suited to building them. Its strong community, encouragement of idiomatic style, and statically- linked binary artifacts make integrating it with other technologies and managing microservices at scale consistent and intuitive. This book will teach you the common patterns and practices, and show you how to apply these using the Go programming language. It will teach you the fundamental concepts of architectural design and RESTful communication, and introduce you to the patterns that provide manageable code that is supportable in development and at scale in production. We will provide you with examples of how to put these concepts and patterns into practice with Go. Whether you are planning a new application or working in an existing monolith, this book will explain and illustrate with practical examples how teams of all sizes can start solving problems with microservices. It will help you understand Docker and Docker Compose, and how they can be used to isolate microservice dependencies and build environments. We will conclude by showing you various techniques to monitor, test, and secure your microservices. By the end, you will know the benefits of the system resilience of a microservice and the advantages of the Go stack. What this book covers Chapter 1 , Introduction to Microservices , looks at what makes the Go language suitable for building microservices and takes a look at the standard library that has all the components required to build a basic microservice. Looking at the standard elements first will give you a grounding and make you appreciate how some of the frameworks that we will discuss later can be incredibly useful. Chapter 2 , Designing a Great API , looks at what makes a good API. We will introduce both REST and RPC, explaining the differences between them. We will also examine the best practices for writing and versioning APIs. Chapter 3 , Introducing Docker , explains how you can wrap your application into a Docker image and how you can use Docker and Docker Compose as part of your development workflow. We will see how it is possible to build a small lightweight image for your application and some good practices for using Docker and writing Dockerfiles. Chapter 4 , Testing , will introduce the various techniques to ensure that your microservices are of the highest quality. We will look at unit testing, behavioral testing, and performance testing, providing you with practical advice and knowledge of the core testing frameworks. Chapter 5 , Common Patterns , introduces some of the standard patterns often employed in microservice architecture. We will take an in-depth look at load balancing, circuit breaking, service discovery, and the autopilot pattern to see what a Go-specific implementation for this would look like. Chapter 6 , Microservice Frameworks , builds on frameworks that implement many of the common features needed for a microservice. We will compare and contrast these through examples of their usage. Chapter 7 , Logging and Monitoring , examines essential techniques to ensure that your service is behaving correctly, and when it does not, ensures you have all the information at your disposal for successful diagnostics and debugging. We will look at using StatsD for simple metrics and timings, how to handle log file information, and approaches to logging more detailed data and platforms such as NewRelic, which provides a holistic overview of your service. Chapter 8 , Security , takes a look at authentication, authorization, and security for your microservice. We will look at JWT and how you can implement middleware for validating your requests and keeping things secure. We will also look at the bigger picture, looking at why you should implement TLS encryption and a principle of no trust between your services. Chapter 9 , Event-Driven Architecture , discusses that a common pattern to allow your microservices to collaborate using Events; you will learn about two of the most common eventing patterns and see how you can implement them in Go. We will also look at the introduction of Domain-Driven Design and how the use of a ubiquitous language can help your software development process. Chapter 10 , Continuous Delivery , discusses the concepts behind continuous delivery. We will then examine in detail a continuous delivery setup for one of the simple applications we created earlier in the book. What you need for this book Go compiler for running your Go codes successfully. You can find it on https://golang.org You also need Docker for container applications. Dockers are available on https://www.docker.com/