[{"content":"DNS Service? What DNS Service? Whilst it has never been announced or details ever posted, BITServices does host a fully public DNS over TLS, DNS over HTTPS and DNS over QUIC service. The service was intended for internal use mainly. When trying to use such a service when out and about however, it really needs to be fully open to the public Internet.\nAbout the BITServices DNS Service The DNS service is as simple as possible and it only serves a single purpose: to adblock based on Steven Black\u0026rsquo;s blocklist.\nAny request that is not blocked is forwarded (over TLS) to CloudFlare\u0026rsquo;s 1.1.1.1.\nDescribe Public? The article about the public DNS server seems to be missing something obvious, the actual service address.\nWell it was never meant to go public in that way. It was meant for internal use mainly so the addresses were never handed out. There was however an expectation that people may stumble across it and use it. As long as the current hosting size can handle the load, it is absolutely fine.\nRecently, it seems the addresses have been discovered (I am not posting where). This has led to a noticeable uptick in use. Whilst everything is still well within limits today, it will be stated here that if that changes, network blocking will be implemented.\nIt also means that seemingly silly changes can be implemented at will, at any time, without warning. With that in mind\u0026hellip;\nIPv6 \u0026hellip; in the near future, the DNS service will be removed from the IPv4 Internet entirely and will be accessible by IPv6 only.\nThis is NOT a blocking measure. The service is hosted on AWS and they have now implement costs for IPv4 addresses. As an IPv6 evangelist myself, every use case I have for the DNS service now has access to the IPv6 Internet. With that in mind, given the purpose of the DNS service, it makes sense to reduce cloud costs.\nAlmost all random traffic to the DNS service appears to be IPv4. This post is mainly to help anyone who wonders what happened to the DNS service when the move to IPv6 only happens.\nTraffic is IPv4? How do you know?! So this might raise questions around logging. The short answer is that NO, the DNS servers do NOT log requests.\nThat said, it is not a promise, or a guarantee. When messing around, trying stuff, etc, logging can be temporarily enabled to assist with this. There is no development environments, so all messing around is carried out in production. The last time I happened to do this, I noticed a LOT of IPv4 traffic.\nI Want One! Want your own ad blocking encrypted DNS server?\nPacker The Packer repository is fully open and this builds an AWS image to launch the DNS server from.\ndns-aws\nCertificates To set up TLS, you\u0026rsquo;d need to use something like CertBot, which is easy if you run a single server, but more interesting if you run a couple (or more).\nI use these helper scripts: bitservices-letsencrypt\n","date":"2025-03-04T12:54:00Z","permalink":"https://www.bitservices.io/blog/dns-server/","title":"The BITServices Public DNS Service"},{"content":"Introduction One really underused feature in platform engineering in my experience is the Language Server Protocol (LSP).\nMore traditional software developers may not get as much value, as they tend to use more full fat Integrated Development Environments (IDEs) such as Visual Studio, IntelliJ IDEA or PhpStorm depending on the specific language a project is developed in.\nPlatform engineers (or \u0026lsquo;DevOps\u0026rsquo; engineers) tend to use more lightweight editors due to the amount of different languages and/or markups they need to work with often within the same project. Many of these languages, such as HCL for developing infrastructure as code in Terraform, do not have a corresponding full fat IDE. Platform engineers would normally use something like Visual Studio Code or Vim.\nLSP enables lightweight editors to get full fat IDE features for many different languages and even markup languages such as YAML. Since LSP decouples the editor from the language tools it allows you to get the same rich features no matter what your editor of choice is (as long as it supports LSP!).\nSince I have spent many years living in Terraform, YAML, shell scripts, etc the prospect of having features such as code completion, diagnostics and automatic formatting for these languages in my favourite editor was very exciting!\nLanguage Server Protocol Overview LSP was originally developed by Microsoft.\nOfficial LSP page: https://microsoft.github.io/language-server-protocol/ List of Language Servers: https://langserver.org/ Without Language Server Protocol Without Language Server Protocol\nEach IDE needed its own code processing engine in order to offer rich development features.\nWith Language Server Protocol With Language Server Protocol\nMany different editors can use multiple stand-alone code processing engines that can be maintained seperately.\nExamples In these example Neovim is used.\nGo Go with Language Server Protocol\nGo without Language Server Protocol\nBicep Bicep with Language Server Protocol\nBicep without Language Server Protocol\nPowershell Powershell with Language Server Protocol\nPowershell without Language Server Protocol\nYAML YAML with Language Server Protocol\nYAML without Language Server Protocol\nWhat is particularly impressive with the YAML language server is that it can tell you are working with certain types of files and can do code completetion and diagnostics against a known schema. In this example, a Gitlab CI file is used and you can see that the Language Server is working with a schema for Gitlab CI. Closing Notes Since using LSP the amount of bad commits I made has been drastically reduced since I can see syntax and schema errors infront of me as I make them. Since formatting is also taken care of automatically this reduces excess commits later on during the pull request process.\nI do not use VSCode so sadly do not know how to get LSP going there if it does not do it out the box. A cursory search reveals little. I am certain that VSCode does support LSP.\nI am not sure if Vim supports LSP natively yet. I personally use Neovim which does support LSP natively. The following link has details that help setting up Neovim: https://github.com/neovim/nvim-lspconfig.\nOften people have compared LSP to Github Copilot. In my view they are very different things. Copilot can use AI to suggest code and even entire functions. LSP on the other hand provides direct language specific tooling. LSP will show suggestions based on what methods, variables, etc are available whereas Copilot will try and predict what you are trying to code by for example the name given to a function.\n","date":"2023-02-22T16:15:00+01:00","permalink":"https://www.bitservices.io/blog/language-server-protocol/","title":"Language Server Protocol (LSP)"},{"content":"\nA part time project where the main goal was to implement Azure Front Door into the existing infrastructure automation to improve security and performance of an application with hundreds of tenants.\nAzure Front Door Performance improvements were achieved by leveraging caching, compression and the content delivery network to service static and cached content from regional distribution points.\nTo improve security Azure Front Door Web Application Firewall (WAF) was used to identify improvements in the application such as ways to filter/encode data that was transmitted to/from the application and customers web browsers. The WAF also enabled visibility to how the site was currently being scraped/attacked and potential mitigation.\nTechnologies Used The existing infrastructure automation was primarily in Bicep and Powershell. These technologies have not previously been used by BITServices and therefore brought new knowledge to the company and contrast with the alternative tools that we much more commonly use.\nOther Deliverables As well as delivering Azure Front Door, other small tasks carried out included creating another environment with infrastructure as code which required heavy refactoring. The refactoring was mainly converting scripted PowerShell steps into proper infrastructure as code (Bicep in this case) and fixing infrastructure deployment pipelines that were failing for various reasons.\n","date":"2022-08-01T00:00:00Z","permalink":"https://www.bitservices.io/projects/smartr365/","title":"Smartr365"},{"content":"\nFor Arden, there were two projects to deliver: an Amazon Elastic Kubernetes Service (EKS) based platform and development of their serverless integrations.\nIn order to help with the delivery of these projects, some open source contributions were made.\nElastic Kubernetes Service (EKS) Platform A new platform was required to migrate existing on-premises and Amazon EC2 workloads to. The new platform had to allow developers to easily manage their own deployments, allow for centralised logging and monitoring, had to be secure and allow for management of things such as DNS records and TLS certificates to be fully automated.\nThe platform had to be delivered \u0026lsquo;as code\u0026rsquo; so it could be easily managed, upgraded and replicated. This was done using mainly Terraform with some base Kubernetes services being deployed using the Terraform Helm provider.\nAny base Kubernetes services that ArgoCD did not depend on explicitly were managed and deployed by ArgoCD along with the business applications hosted on the platform.\nManaging Deployments with ArgoCD The main interface into the platform for developers was through ArgoCD. This allowed developers to deploy applications, see the status of their workloads and get immediate logs and events to diagnose failures. Being able to deploy and diagnose simple issues themselves allowed developers to spend less time being blocked waiting for infrastructure engineers to help out.\nArgoCD was integrated with Microsoft Entra to allow single-sign on which improves security and convenience. Developers could be assigned to individual projects meaning they only have access to the applications that they are working with.\nArgoCD Image Updater was also used in development environments so that successful builds of an application could be deployed automatically.\nMonitoring \u0026amp; Logging For the monitoring and logging, open source tools were used as they provide excellent flexibility and of course value.\nFor monitoring, Prometheus was used with Grafana. Prometheus was also configured to make a future expansion to Thanos as easy as possible, should the need ever arise.\nLoki was used for logging, it integrated perfectly with the monitoring metrics since it also uses Grafana as the main user interface.\nAlerts were configured with Grafana unified alerts, allowing alerts to be raised from either metric or logging data.\nAutomated Management Tools such as Certificate Manager and External DNS were used to allow automation of day to day administrative tasks.\nCertificate Manager automatically provisions and renews TLS certificates for applications hosted within the platform whilst External DNS creates and cleans up DNS records.\nFuture Ready Whilst not immediately implemented due to possible conflicts with the business the platform was developed and tested with IPv6 support and support for ARM based compute allowing for easy adoption later on.\nServerless Integrations Development Arden was implementing a new student record system called SITS. This was a huge project that affected all areas of the business. A significant problem was integrating existing systems that do not understand SITS, such as: the virtual learning environment, Active Directory, time tabling products and many more.\nTo enable this communication between systems, a group of serverless \u0026lsquo;middlewares\u0026rsquo; were used to keep things scalable and event driven.\nSince the integrated systems spanned across AWS and Azure as well as including software as a service (SaaS) solutions, many technologies were used to integrate them. These technologies included: Typescript based Lambda functions deployed with the Amazon CDK, Azure Logic Apps, Azure Functions, Amazon SNS, Amazon SQS and Azure Service Bus.\nThese integrations handled large amounts of traffic especially as SITS went live at Arden. They also helped the business implement complex processes with systems that cannot be directly integrated.\nOpen Source Contributions The following open source contributions were made whilst delivering these projects:\nPrometheus: discovery: Allow EC2 Service Discovery to work with IPv6-only instances External DNS: feat(aws): always create AAAA alias records in route53 ms_active_directory: Multiple contributions onedark.nvim: Fix NvimTree Floating Windows lazygit.nvim: feat: Allow current buffer commits to show on the already worked out Git root ","date":"2022-01-01T00:00:00Z","permalink":"https://www.bitservices.io/projects/arden/","title":"Arden University"},{"content":"Introduction There are many guides on the Internet for getting started with Terraform and even setting up remote state. There seems however to be very few (if any) that suggest ways of setting it up in a scalable way. Normally, in such guides the remote state is statically declared in the Terraform code. This forbids the code to be easily re-used for different environments without duplicating it all.\nThe purpose of this post is to put forward some ways that I have used myself or seen used over the past few years.\nThis post will focus on the AWS S3 remote backend, but the concepts will apply to others as well, such as the Azure storage account (azurerm) backend.\nStarting Point Both Terraform and the AWS CLI tool are installed.\nAn S3 bucket exists that can be used for Terraform remote state.\nThe current AWS CLI tool profile has read and write access to the Terraform remote state bucket.\nThere is some Terraform code that needs to be deployed to multiple environments. In this example the following code is used as a starting point. Everything is in a single file to make the example more simple and the remote backend is statically defined, as per most examples on the Internet:\ns3.tf ############################################################################### terraform { required_version = \u0026#34;\u0026gt;= 1.0.0\u0026#34; backend \u0026#34;s3\u0026#34; { key = \u0026#34;object-store/terraform.state\u0026#34; bucket = \u0026#34;example-terraform-state-bitservices\u0026#34; region = \u0026#34;eu-west-1\u0026#34; encrypt = true } required_providers { aws = { source = \u0026#34;hashicorp/aws\u0026#34; } } } ############################################################################### provider \u0026#34;aws\u0026#34; { region = \u0026#34;eu-west-1\u0026#34; } ############################################################################### variable \u0026#34;account\u0026#34; { default = \u0026#34;bitservices\u0026#34; } variable \u0026#34;environment\u0026#34; { default = \u0026#34;default\u0026#34; } ############################################################################### variable \u0026#34;encryption_key\u0026#34; { default = null } variable \u0026#34;encryption_type\u0026#34; { default = \u0026#34;AES256\u0026#34; } ############################################################################### variable \u0026#34;acl\u0026#34; { default = \u0026#34;private\u0026#34; } variable \u0026#34;service\u0026#34; { default = \u0026#34;object-store\u0026#34; } variable \u0026#34;force_destroy\u0026#34; { default = false } ############################################################################### locals { name = format(\u0026#34;%s-%s-%s\u0026#34;, var.service, var.environment, var.account) encryption_key = var.encryption_type == \u0026#34;aws:kms\u0026#34; ? var.encryption_key : null } ############################################################################### resource \u0026#34;aws_s3_bucket\u0026#34; \u0026#34;scope\u0026#34; { acl = var.acl bucket = local.name force_destroy = var.force_destroy tags = { Name = local.name Account = var.account Service = var.service Environment = var.environment } dynamic \u0026#34;server_side_encryption_configuration\u0026#34; { for_each = lower(var.encryption_type) == \u0026#34;none\u0026#34; ? [] : tolist([var.encryption_type]) content { rule { apply_server_side_encryption_by_default { sse_algorithm = server_side_encryption_configuration.value kms_master_key_id = local.encryption_key } } } } } ###############################################################################\nPlease note: This example does not include state locking or the use of Terraform modules to try and keep the post as on-topic and as short as possible.\nOption 1: Workspaces One of the simplest ways to make some Terraform code re-usable across different environments is to use Terraform Workspaces.\nThe easiest way to use workspaces in this way is to ensure each resource identifier includes the workspace name. Settings, for example instance sizes or if encryption is to be enabled or not can be defined in maps with the workspace names as keys. This allows settings to be looked up based on the currently enabled workspace.\nThe example Terraform code above modified to work with multiple workspaces could look something like this:\n############################################################################### terraform { required_version = \u0026#34;\u0026gt;= 1.0.0\u0026#34; backend \u0026#34;s3\u0026#34; { key = \u0026#34;object-store/terraform.state\u0026#34; bucket = \u0026#34;example-terraform-state-bitservices\u0026#34; region = \u0026#34;eu-west-1\u0026#34; encrypt = true workspace_key_prefix = \u0026#34;object-store-env\u0026#34; } required_providers { aws = { source = \u0026#34;hashicorp/aws\u0026#34; } } } ############################################################################### provider \u0026#34;aws\u0026#34; { region = \u0026#34;eu-west-1\u0026#34; } ############################################################################### variable \u0026#34;account\u0026#34; { default = \u0026#34;bitservices\u0026#34; } ############################################################################### variable \u0026#34;encryption_key\u0026#34; { default = null } variable \u0026#34;encryption_type\u0026#34; { default = { \u0026#34;default\u0026#34; = \u0026#34;none\u0026#34;, \u0026#34;prod\u0026#34; = \u0026#34;AES256\u0026#34; }} ############################################################################### variable \u0026#34;acl\u0026#34; { default = \u0026#34;private\u0026#34; } variable \u0026#34;service\u0026#34; { default = \u0026#34;object-store\u0026#34; } variable \u0026#34;force_destroy\u0026#34; { default = false } ############################################################################### locals { name = format(\u0026#34;%s-%s-%s\u0026#34;, var.service, local.environment, var.account) environment = terraform.workspace encryption_key = var.encryption_type == \u0026#34;aws:kms\u0026#34; ? var.encryption_key : null encryption_type = lookup(var.encryption_type, local.environment, \u0026#34;none\u0026#34;) } ############################################################################### resource \u0026#34;aws_s3_bucket\u0026#34; \u0026#34;scope\u0026#34; { acl = var.acl bucket = local.name force_destroy = var.force_destroy tags = { Name = local.name Account = var.account Service = var.service Environment = local.environment } dynamic \u0026#34;server_side_encryption_configuration\u0026#34; { for_each = lower(local.encryption_type) == \u0026#34;none\u0026#34; ? [] : tolist([local.encryption_type]) content { rule { apply_server_side_encryption_by_default { sse_algorithm = server_side_encryption_configuration.value kms_master_key_id = local.encryption_key } } } } } ############################################################################### Please note: normally it is best to use encryption for all environments. This was just changed for the purpose of being an example.\nMake sure Terraform has been initialised: $ terraform init Initializing the backend... Successfully configured the backend \u0026#34;s3\u0026#34;! Terraform will automatically use this backend unless the backend configuration changes. Initializing provider plugins... ... Then, to create a new workspace for production, run: $ terraform workspace new prod Created and switched to workspace \u0026#34;prod\u0026#34;! You\u0026#39;re now on a new, empty workspace. Workspaces isolate their state, so if you run \u0026#34;terraform plan\u0026#34; Terraform will not see any existing state for this configuration Now when running a plan or apply, a new, unique S3 bucket will be managed.\nUsing workspaces still uses a statically defined remote backend key. Each non-default workspace state file will have an automatically appended prefix.\nOption 2: Simple Wrapper Script Another way of splitting out remote state can be to mirror the remote storage backend with the local file system. This is more useful if there are lots of different pieces of Terraform within one repository but there are not multiple environments for each individual piece.\nFor this to work with our example we would have to do the following:\nTurn our workspace into a Git repository, if it is not already one: $ git init Move our s3.tf file into its own folder. This is based on the original s3.tf file and NOT the one modified to work with workspaces: $ mkdir s3 $ mv ./s3.tf ./s3/s3.tf Remove the following line from the backend configuration in our s3.tf file, since we will be generating it with scripts: key = \u0026#34;object-store/terraform.state\u0026#34; Create a common.sh file with the content below. This file should NOT be executable as it will only ever be sourced: common.sh ############################################################################### if [[ \u0026#34;${BASH_SOURCE[0]}\u0026#34; == \u0026#34;${0}\u0026#34; ]]; then echo \u0026#34;Please do not run this file directly!\u0026#34; exit 1 fi ############################################################################### TF_BASE=\u0026#34;$(git rev-parse --show-toplevel)\u0026#34; ############################################################################### if [ -n \u0026#34;${TF_PREFIX}\u0026#34; ] \u0026amp;\u0026amp; [ -d \u0026#34;${TF_PREFIX}\u0026#34; ]; then cd \u0026#34;${TF_PREFIX}\u0026#34; TF_PATH=\u0026#34;$(pwd -P)\u0026#34; else echo \u0026#34;Error: No Terraform folder specified or folder does not exist!\u0026#34; exit 1 fi ############################################################################### if [[ \u0026#34;${TF_PATH}\u0026#34; != \u0026#34;${TF_BASE}\u0026#34;* ]]; then echo \u0026#34;Error: The folder given does not exist within the Git repository.\u0026#34; exit 1 fi ############################################################################### S3_STATE_FILENAME=\u0026#34;terraform.tfstate\u0026#34; S3_STATE_KEY=\u0026#34;$(git rev-parse --show-prefix)${S3_STATE_FILENAME}\u0026#34; ############################################################################### TF_VAR_base=\u0026#34;${TF_BASE}\u0026#34; TF_VAR_path=\u0026#34;${TF_PATH}\u0026#34; ############################################################################### terraform --version ############################################################################### echo \u0026#34;\u0026#34; echo \u0026#34;:: Base : ${TF_BASE}\u0026#34; echo \u0026#34;:: Path : ${TF_PATH}\u0026#34; echo \u0026#34;:: S3 Key: ${S3_STATE_KEY}\u0026#34; echo \u0026#34;\u0026#34; ################################################################################ terraform init --input=false --backend=true --backend-config=\u0026#34;key=${S3_STATE_KEY}\u0026#34; ################################################################################\nCreate a plan.sh file with the below content. This file SHOULD be executable as it will be directly called to do a Terraform plan: plan.sh #!/bin/bash -e ############################################################################### set -o pipefail ############################################################################### TF_PREFIX=\u0026#34;${1}\u0026#34; ############################################################################### source \u0026#34;./common.sh\u0026#34; ############################################################################### terraform plan ###############################################################################\nBased on plan.sh, create apply.sh, destroy.sh, etc. From here the shell scripts are used to call Terraform and set the remote state key based on what local folder we are running Terraform against.\nFor example: $ ./apply.sh s3 Will create our S3 bucket and put the remote state in the following S3 key: s3/terraform.tfstate.\nIf we created another folder called ec2 that had code to create an EC2 instance and called it with the same scripts: $ ./apply.sh ec2 Will create the EC2 instance and put the remote state in the following S3 key: ec2/terraform.tfstate.\nWhilst fairly simple this approach does have some drawbacks:\nManagement of a small shell script for each Terraform sub-command.\nDifficult to manage environments for the same pieces of Terraform code without complex use of symlinks.\nOption 3: Full Wrapper Library Ultimately this is where I have ended up and many organisations that use Terraform extensively are also likely to end up. A Terraform wrapper can be created to not only organise the remote state storage, it could manage Terraform binary versions, manage authentication with the cloud provider, do a degree of configuration management, make calling from CI or locally the same and any other organisation specific things.\nA wrapper can be created in any language, though it can be nice to have it integrate with a build system like GNU Make or Rake so calling Terraform and non-Terraform tasks feel the same.\nThe Terraform wrapper I use and maintain is located: https://rubygems.org/gems/terraform-wrapper. Sadly it is not yet documented at this stage.\nThis wrapper integrates with the Rake build system and provides Terraform related tasks to multiple folders containing Terraform infrastructure.\n$ rake -T [I] [TerraformWrapper] Terraform Wrapper for Ruby - version: 1.2.0 [I] [TerraformWrapper] Building tasks for service: account, component: bootstrap... [I] [TerraformWrapper] Building tasks for service: account, component: account... rake account:apply[config,plan] # Applies infrastructure with Ter... rake account:binary # Downloads and extracts the expe... rake account:clean # Cleans a Terraform infrastructu... rake account:destroy[config] # Destroys infrastructure with Te... rake account:import[config,address,id] # Import a piece of existing infr... rake account:init[config] # Initialises the Terraform infra... rake account:plan[config,out] # Creates a Terraform plan for a ... rake account:plan-destroy[config,out] # Creates a Terraform destroy pla... rake account:upgrade # Upgrades the Terraform infrastr... rake account:validate # Validates the Terraform code fo... rake bootstrap:apply[config,plan] # Applies infrastructure with Ter... rake bootstrap:binary # Downloads and extracts the expe... rake bootstrap:clean # Cleans a Terraform infrastructu... rake bootstrap:destroy[config] # Destroys infrastructure with Te... rake bootstrap:import[config,address,id] # Import a piece of existing infr... rake bootstrap:init[config] # Initialises the Terraform infra... rake bootstrap:plan[config,out] # Creates a Terraform plan for a ... rake bootstrap:plan-destroy[config,out] # Creates a Terraform destroy pla... rake bootstrap:upgrade # Upgrades the Terraform infrastr... rake bootstrap:validate # Validates the Terraform code fo... Another example of a Terraform wrapper I have seen used and works well is located: https://rubygems.org/gems/rake_terraform.\n","date":"2021-07-31T13:40:22+01:00","permalink":"https://www.bitservices.io/blog/terraform-remote-state/","title":"Scalable Ways to Manage Terraform Remote State"},{"content":"\nA project to pick up support for and improve an existing Kubernetes based infrastructure platform hosted on Microsoft Azure.\nSkip to Result\nStabilising Platform Initially, the primary focus had been to automate continual time consuming tasks and to reduce the number of support tickets. This has been achieved by implementing technologies such as External DNS and Certificate Manager. Reliability has been improved by consolidating and improving complex build pipelines by utilising YAML libraries for Azure DevOps and introducing infrastructure as code with Terraform.\nEvolving Platform After stabilising the platform focus shifted to evolving the platform to be more performant, scalable and developer friendly. In addition to everything outlined below, lots of work went into upgrades of Kubernetes clusters and services, improvements as and when they were identified and migrations to new Azure subscriptions/tenants.\nDeveloper Friendly To help improve continuous delivery, ArgoCD was implemented. This allowed developers to publish Docker images in their build pipelines which would then automatically be picked up and deployed into a Kubernetes environment. ArgoCD also allowed the configuration for the application to be completely separated from other infrastructure as code meaning developers could have full control over the configuration of their deployed applications without requiring permissions to alter other parts of infrastructure.\nInfrastructure as Code There was a large focus on infrastructure as code. Terraform was used to automate the creation of all base infrastructure: such as resource groups, virtual networks, security groups and Kubernetes clusters - including all supporting infrastructure and Active Directory objects. This allowed much quicker and reliable provision of core infrastructure.\nAny new services deployed were also fully automated so that required databases, Kubernetes permissions and the application themselves could also be deployed in a reliable and repeatable way.\nMonitoring \u0026amp; Logging As the platform grew it became important to be able to monitor workloads. Initially Prometheus was used with Grafana as well as some other custom exporters to get certain metrics that were otherwise unavailable.\nHowever as things grew Prometheus was not scaling so eventually the monitoring stack was upgraded with Thanos. This allowed gathering of huge amounts of metrics from multiple Prometheus instances and storing the data in cost efficient blob storage. Thanos also allowed a \u0026lsquo;single pane of glass\u0026rsquo; view of the entire estate of multiple clusters through Grafana.\nCentralised logging was also set up utilising ElasticSearch at first and later New Relic.\nIngress A lot of work was carried out around ingress to the platform. Initially cluster ingresses were standardised with Ingress Nginx. Security and CDN capability was then added using Cloudflare and Cloudflare Access. This allowed many apps to use single sign-on that originally did not support it. It also enabled zero trust access to internal services without having to set up and maintain a VPN.\nResult Due to multiple reasons - the good work that Drivvn do, the pandemic and just the way things are going - significantly more people are now buying cars online. Making the infrastructure more scalable and resilient has helped make it possible to meet these suddenly increasing demands.\nHaving the infrastructure as code ready allowed us to work with one of the Drivvn development teams to deliver a new product from inception to launch within a few weeks. The initial infrastructure was provisioned and available within hours. We then worked with the team to refine the build pipelines and infrastructure as they developed the product. Since launch, the infrastructure supporting this product has been extremely reliable. Having ArgoCD set up allowed developers to manage their own configuration and \u0026lsquo;self-serve\u0026rsquo; deployments all the way to production - with easy roll backs if required.\n","date":"2020-07-02T00:00:00Z","permalink":"https://www.bitservices.io/projects/drivvn/","title":"Drivvn"},{"content":"Introduction Whilst using Kubernetes over the past few months, one challenge I repeatedly faced was to get secrets - such as passwords, SSH keys or certificate keys - securely into applications running on Kubernetes.\nWhilst this is quite easy if the container image is under your full control, to achieve this with an \u0026lsquo;off the shelf\u0026rsquo; image is a little more tricky.\nOne tool I came across recently was confd - which has helped a lot with this challenge and below I will outline how.\nconfd Basics confd is a tool for rendering configuration files from predefined templates using values (secrets) that are stored in a backend. A backend could be etcd, Amazon SSM Parameter Store, Hashicorp Vault or many others.\nThe examples below will be using the Amazon SSM Parameter Store backend. For Kubernetes clusters running on AWS this works really well as Amazon IAM roles can be used, mitigating the use for storing the backend password anywhere.\nI won\u0026rsquo;t go into too much detail on the basics of confd. I would recommend you look at the below links:\nconfd Github Page confd Quick Start Guide confd Image First a Docker container that has confd available will be required. At the time of writing I could not find an official image available - so baked my own. This should be a simple and small image, based on something like Alpine Linux with only confd installed and not much else.\nSince creating this guide, many better ways of handling secrets in Kubernetes are available such as CSI drivers and the use of Terraform with Kubernetes secrets. Due to this and the fact confd has not had a release since 2018 the pre-baked images are no longer available. Example 1: Injecting Secrets into Environment Many \u0026lsquo;off the shelf\u0026rsquo; images allow for loading secrets from environment variables. One example of this is Grafana.\nStarting Example Lets start with injecting secrets as simply as possible - plain text in the deployment spec:\n- name: grafana imagePullPolicy: IfNotPresent image: \u0026#34;grafana/grafana:latest\u0026#34; env: - name: GF_SECURITY_ADMIN_USER value: admin - name: GF_SECURITY_ADMIN_PASSWORD value: supersecurepassword123 We want the Grafana image to get the secrets above, by itself, without having to manage the Grafana image ourselves.\nAdd the Secrets to Amazon SSM Parameter Store Add the two secrets to the Amazon SSM Parameter store using the AWS console.\n/grafana-username: the Grafana administrators username /grafana-password: the Grafana administrators password Create an Amazon IAM Role To allow the containers to access the SSM Parameters, they need to be granted access by IAM.\nIn addition, access to decrypt using the Amazon KMS key used to encypt the parameters will also need to be granted.\nExample (do not copy and paste!):\n{ \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [ { \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;: [ \u0026#34;ssm:GetParameter\u0026#34; ], \u0026#34;Resource\u0026#34;: [ \u0026#34;arn:aws:ssm:eu-west-1:123456123:parameter/grafana-username\u0026#34;, \u0026#34;arn:aws:ssm:eu-west-1:123456123:parameter/grafana-password\u0026#34; ] }, { \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;: [ \u0026#34;kms:Decrypt\u0026#34; ], \u0026#34;Resource\u0026#34;: \u0026#34;arn:aws:kms:eu-west-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab\u0026#34; } ] } confd Configurations confd uses TOML configuration files to define what you want it to process. The below TOML will process template grafana.env.tmpl (defined later) and put the output in /shared-config/grafana.env with mode 0400. As Grafana by default runs as UID:GID 472:472 we make sure the environment file is owned by the same user \u0026amp; group.\napiVersion: v1 kind: ConfigMap metadata: name: grafana-confd-configs namespace: monitoring data: grafana.env.toml: | [template] src = \u0026#34;grafana.env.tmpl\u0026#34; dest = \u0026#34;/shared-config/grafana.env\u0026#34; uid = 472 gid = 472 mode = \u0026#34;0400\u0026#34; keys = [ \u0026#34;/grafana-username\u0026#34;, \u0026#34;/grafana-password\u0026#34; ] confd Templates The templates are the configuration files to render. As we want to set environment variables, the following template works well:\napiVersion: v1 kind: ConfigMap metadata: name: grafana-confd-templates namespace: monitoring data: grafana.env.tmpl: | export GF_SECURITY_ADMIN_USER=\u0026#34;{{ getv \u0026#34;/grafana-username\u0026#34; }}\u0026#34; export GF_SECURITY_ADMIN_PASSWORD=\u0026#34;{{ getv \u0026#34;/grafana-password\u0026#34; }}\u0026#34; Override Launcher A new launcher script should be created for the main container (Grafana in this example). The new launcher script should import rendered environment variables and then start the original entrypoint script.\nPlease note the following:\nAlways use . instead of source. A lot of containers do not have a full bash shell. Always exec to start the original entry point - so that it remains as PID 1. /run.sh is the original entry point of the Grafana image. Make sure \u0026quot;${@}\u0026quot; is passed to the original entry point, so arguments still work. To find the original entry point of an image, download the image with docker pull and then use docker inspect to find the entry point.\napiVersion: v1 kind: ConfigMap metadata: name: grafana-launcher namespace: monitoring data: launcher.sh: | #!/bin/bash -e ############################################################################### echo \u0026#34;:: Loading extra environment variables...\u0026#34; . \u0026#34;/shared-config/grafana.env\u0026#34; ############################################################################### echo \u0026#34;:: Launching Grafana...\u0026#34; exec \u0026#34;/run.sh\u0026#34; \u0026#34;${@}\u0026#34; ############################################################################### Modifying the Deployment The final step is to make the Grafana deployment run confd based on the supplied configuration before Grafana is started. To do that we use an initContainer.\nNotice:\nWe set the region to eu-west-1, but you need to set this to the region your SSM parameters are stored. We have three volume mounts: grafana-shared-config is a shared emptyDir volume for the main Grafana container and the confd initContainer. grafana-confd-configs will refer to the confd configurations configuration map defined above. grafana-confd-templates will refer to the confd templates configuration map defined above. initContainers: - name: grafana-confd image: \u0026#34;rlees85/secrets-loader:latest\u0026#34; command: [ \u0026#39;confd\u0026#39;, \u0026#39;-onetime\u0026#39;, \u0026#39;-backend\u0026#39;, \u0026#39;ssm\u0026#39; ] env: - name: AWS_DEFAULT_REGION value: eu-west-1 volumeMounts: - name: grafana-shared-config mountPath: /shared-config - name: grafana-confd-configs mountPath: /etc/confd/conf.d - name: grafana-confd-templates mountPath: /etc/confd/templates The grafana-shared-config and grafana-launcher mount should be added to the main Grafana container.\n- name: grafana-shared-config mountPath: /shared-config - name: grafana-launcher mountPath: /launcher All volumes should be correctly defined in the deployment. Please note that in the particular deployment used in this example grafana-config was already present.\nvolumes: - name: grafana-config configMap: name: grafana - name: grafana-shared-config emptyDir: {} - name: grafana-confd-configs configMap: defaultMode: 0400 name: grafana-confd-configs - name: grafana-confd-templates configMap: defaultMode: 0400 name: grafana-confd-templates - name: grafana-launcher configMap: defaultMode: 0500 name: grafana-launcher The initContainer needs access to Amazon SSM Parameter store. Make sure kube2iam is configured on the Kubernetes Cluster and add the appropriate annotation to the Grafana deployment.\nannotations: iam.amazonaws.com/role: grafana Finally, we can override the Grafana containers start-up command to use the new launcher script:\n- name: grafana imagePullPolicy: IfNotPresent image: \u0026#34;grafana/grafana:latest\u0026#34; command: [ \u0026#39;/launcher/launcher.sh\u0026#39; ] Conclusion The confd initContainer now runs before Grafana starts and outputs the templated secrets to shared storage. The main Grafana container then sources these secrets from shared storage before running the original image entry point.\n$ kubectl -n monitoring logs grafana-7646488856-4f4gx -c grafana-confd 2018-08-06T19:34:55Z grafana-7646488856-4f4gx confd[1]: INFO Backend set to ssm 2018-08-06T19:34:55Z grafana-7646488856-4f4gx confd[1]: INFO Starting confd 2018-08-06T19:34:55Z grafana-7646488856-4f4gx confd[1]: INFO Backend source(s) set to 2018-08-06T19:34:56Z grafana-7646488856-4f4gx confd[1]: INFO Target config /shared-config/grafana.env out of sync 2018-08-06T19:34:56Z grafana-7646488856-4f4gx confd[1]: INFO Target config /shared-config/grafana.env has been updated $ kubectl -n monitoring logs grafana-7646488856-4f4gx t=2018-08-06T19:35:07+0000 lvl=info msg=\u0026#34;Starting Grafana\u0026#34; logger=server version=5.2.1 commit=2040f61 compiled=2018-06-29T09:17:46+0000 ... t=2018-08-06T19:35:07+0000 lvl=info msg=\u0026#34;Config overridden from Environment variable\u0026#34; logger=settings var=\u0026#34;GF_SECURITY_ADMIN_USER=admin\u0026#34; t=2018-08-06T19:35:07+0000 lvl=info msg=\u0026#34;Config overridden from Environment variable\u0026#34; logger=settings var=\u0026#34;GF_SECURITY_ADMIN_PASSWORD=*********\u0026#34; ... Example 2: Rendering Configuration Files and/or Keys Please read through example 1 first. A lot of things will not be covered again and are assumed to be already set up (SSM parameters, KMS keys and IAM permissions).\nIn this example we have a much more complicated application, that requires secrets to be loaded into its configuration files. Additionally, the application integrates with other services - and therefore needs an SSH private key to be injected at run-time.\nStarting Example In this example, the deployment spec has no secrets in. The secrets are baked directly in to the image. This may be undesirable for example if the image has to pass through a pipeline - developers perhaps should not have access to production secrets.\nLet\u0026rsquo;s say the following file is baked directly into the image:\n$ cat /etc/application.d/50-config.properties mysql.db.username=application mysql.db.password=application123 integration.ssh-key=/etc/application/ssh.pem NOTE: When loading multi-line parameters (such as SSH keys) into Amazon SSM Parameter store use the CLI tool and not the console! If the console is used new lines are lost.\nconfd Configurations In a similar fashion to the first example, we template a configuration file and SSH key based on templates to shared storage.\napiVersion: v1 kind: ConfigMap metadata: name: application-confd-configs data: 99-secrets.properties.toml: | [template] src = \u0026#34;99-secrets.properties.tmpl\u0026#34; dest = \u0026#34;/shared-config/99-secrets.properties\u0026#34; mode = \u0026#34;0400\u0026#34; keys = [ \u0026#34;/application-db-username\u0026#34;, \u0026#34;/application-db-password\u0026#34; ] integration-key.pem.toml: | [template] src = \u0026#34;integration-key.pem.tmpl\u0026#34; dest = \u0026#34;/shared-config/integration-key.pem\u0026#34; mode = \u0026#34;0400\u0026#34; keys = [ \u0026#34;/application-integration-key\u0026#34; ] confd Templates As before, the templates referred to by the TOML configurations are defined below:\napiVersion: v1 kind: ConfigMap metadata: name: application-confd-templates data: 99-secrets.properties.tmpl: | mysql.db.username={{ getv \u0026#34;/application-db-username\u0026#34; }} mysql.db.password={{ getv \u0026#34;/application-db-password\u0026#34; }} integration.ssh-key=/shared-config/integration-key.pem integration-key.pem.tmpl: | {{ getv \u0026#34;/application-integration-key\u0026#34; }} Override Launcher The same as the first example a new launcher script should be created for the main container. The script should import rendered configuration files into a folder that the application can pick them up.\nThe extra configuration file already points to the rendered SSH key so no further action is required for the key.\napiVersion: v1 kind: ConfigMap metadata: name: application-launcher namespace: monitoring data: launcher.sh: | #!/bin/bash -e ############################################################################### echo \u0026#34;:: Loading extra configuration files...\u0026#34; find \u0026#34;/shared-config\u0026#34; -maxdepth 1 -type f -name \u0026#34;*.properties\u0026#34; -exec cp -sfv {} \u0026#34;/etc/application.d/\u0026#34; \\; ############################################################################### echo \u0026#34;:: Launching Application...\u0026#34; exec \u0026#34;/opt/startup/startup.sh\u0026#34; \u0026#34;${@}\u0026#34; ############################################################################### If the image you are working with does not have find there are many other ways to achieve the same thing.\nModifying the Deployment The deployment needs to be modified the same way as in example 1 above.\nConclusion This shows even complicated configurations can be setup with confd whilst still using off-the-shelf images.\n","date":"2018-08-06T19:22:56+01:00","permalink":"https://www.bitservices.io/blog/confd-kubernetes/","title":"Using confd to Inject Secrets into Kubernetes Pods"},{"content":"\nBITServices Ltd have officially joined the SAP PartnerEdge Open Ecosystem.\nThis allows us access to specialist resources to further help us carry out projects such as cloud migrations for our customers who use the SAP Commerce e-commerce platform.\n","date":"2017-12-12T21:43:04Z","permalink":"https://www.bitservices.io/partners/sap-open-ecosystem/","title":"SAP"},{"content":"\nA very exciting \u0026lsquo;greenfield\u0026rsquo; project creating a new infrastructure platform using Amazon Web Services, Docker and Kubernetes for SAP Commerce based e-commerce websites.\nIt had been a great pleasure working with the team at Eclipse whilst delivering this project.\nSkip to Result\nProject Breifing The project aim was to create an Amazon Web Services based, fully automated infrastructure platform to host SAP Commerce e-commerce websites. The platform must be constructed in a way that it can be used on other cloud service providers with little effort later on.\nDevelopers and testers needed the ability to get code and features through testing pipelines much quicker. In addition, production websites need to handle peak demand seamlessly.\nAs well as delivering a platform, existing teams were to be assisted with picking up new tools, technologies and concepts to enable on-going support of the platform.\nSolution The final solution comprised of many components. These components are outlined below.\nInfrastructure as Code Writing infrastructure as code was key to this solution. This is what enabled infrastructure to be provisioned in a reliable and repeatable way at the click of a button. By taking advantage of Terraform module sources it was possible to define a collection of infrastructure objects (such as subnets, route tables, gateways) in a single place but allow variables (such as name, CIDR ranges) to be passed in depending on the environment being built. As a result all infrastructure met defined standards, human error is vastly reduced and development/production parity was achieved.\nBy taking advantage of Amazon Web Services availability zones and infrastructure as code, all production environments were highly available and could withstand the loss of an Amazon data centre without any downtime.\nImmutable Software To allow software (SAP Commerce in particular) to run in a dynamically scaled environment there were a few challenges to overcome. There needed to be a way to start SAP Commerce very quickly in \u0026lsquo;scale-up\u0026rsquo; situations additionally the state of any running SAP Commerce instances had to be externalised in case of scale-down.\nDocker was chosen to containerise software. Containerisation enabled the application and all dependencies, configurations, etc to be packed into an image that can be started very quickly. By using SAP Commerce \u0026lsquo;aspects\u0026rsquo; a single Docker image can be capable of running in multiple environments in multiple modes. A single image can be promoted all the way through the testing pipeline just by using tags.\nTo make running containers stateless, services such as Amazon S3 and EFS were used.\nService Orchestration To enable service-level auto scaling, auto healing, multi-tenant clustering and service health checks, Kubernetes was implemented. This allowed the platform to meet the scaling requirements. The healing features of Kubernetes allowed the platform to be more resilient to virtual machine failure or network outages resulting in a higher service availability.\nTechnologies \u0026amp; Tools Multiple technologies and tools were used to deliver this project.\nAmazon Web Services The initial cloud provider chosen was Amazon. Amazon Web Services (AWS) is a mature cloud service with endless offerings. It is effortless to build highly-available and highly-performant infrastructure stacks. Most tools (such as Terraform and Kops below) offer excellent support for AWS. By using the many AWS services available such as IAM and availability zones it was possible to create secure and resilient infrastructure.\nHashicorp Terraform Terraform was chosen to build foundation and networking infrastructure. Terraform has excellent Amazon Web Services support and code can easily be ported to work with other cloud providers too - including OpenStack for managing resources on-premises. Terraform made it very easy to meet the project requirements to stay cloud agnostic and to fully automate infrastructure.\nDocker Containerisation was chosen to help simplify development. Rather than pushing a codebase that may have a complete different set of steps to deploy depending on target environment, containerisation makes that a single image that may take the target environment as a parameter. Additionally, all dependencies and libraries required to run the application are packed into the container, meaning that the same container can be ran locally, on tin or in the cloud with minimal effort. Docker was chosen specifically due to it being well proven and mature.\nKubernetes Kubernetes is fast becoming an industry standard for container orchestration. It can also run on any cloud provider or even on-premises on tin. Native support for service-level auto-scaling and \u0026lsquo;cluster autoscaler\u0026rsquo; add-on for the scaling of underlying virtual machines allowed scaling requirements to be met. The many different types of services (deployments, statefulsets, daemonsets), specifications (disruption budgets, affinities) and probes (liveness, readiness) made it possible to build a platform that is resilient against hypervisor, network or even data centre failures.\nOther Tools Other tools used to deliver this project include: Kops, Hasicorp Packer and Ansible.\nResult The result of this project allows Eclipse to offer their customers an improved hosting service. Development environments can now be provisioned rapidly and easily decommissioned when not in use to match project demands. Automatic scaling allows customers websites to seamlessly handle high load during sales and events whilst running economically during quiet periods. The platform allows for zero-downtime code deployments and platform updates. High availability is achieved by always running across multiple availability zones (data centres) in addition to having automatic health checks and repairs.\n","date":"2017-11-06T00:00:00Z","permalink":"https://www.bitservices.io/projects/eclipse-group-solutions/","title":"Eclipse Group Solutions"},{"content":"\nA small project carried out for a family member.\nThe Problem FoodFriends have a small Wordpress website that was hosted with a traditional hosting provider. The provider, unfortunately had crippling limits such as only 50MiB of disk for website media. FoodFriends required that website stayed on the Wordpress platform.\nThe Solution The website was moved to Amazon Web Services.\nTo remove future support costs and time, the supporting infrastructure was automated with Hashicorp Terraform. The installation and configuration of software fully automated with Ansible - even down to the renewal of the TLS certificate!\nBy using Amazon S3 backups are automatically taken and rotated daily. Restoration of data is managed by the Ansible playbooks meaning the entire infrastructure can be built from nothing with all data restored in just minutes.\nThe Result There is now gigabytes of available diskspace for media. As Amazon Web Services is an enterprise grade platform capable of hosting even the most busiest of e-commerce websites the performance of the FoodFriends website has increased dramatically.\nHaving TLS configured correctly and being made mandatory the FoodFriends website has had its ranking increased on Google.\nThe current cost of hosting the infrastructure is approximately $1 a month for a couple of Route53 zones. Once the free-tier expires, the overall cost will still be only a third of what it was with the previous host.\n","date":"2017-09-01T00:00:00Z","permalink":"https://www.bitservices.io/projects/foodfriends/","title":"FoodFriends"}]