Hjortsberg's notes

Building Nginx modules using CMake

Wed, 28 Dec 2022 00:00:00 +0100

Introduction

Modules in Nginx can be either dynamic or static. A dynamic module is a shared object that is loaded by Nginx at runtime and Nginx need to be instructed in the configuration to load the module.

A static module is compiled into the Nginx binary and, hence, does not need to be specified in Nginx configuration.

With dynamic modules you can usually use the Nginx version provided by the package manager of your operating system so you only build the module and update the configuration.

With a static module you will need to build your own Nginx binary.

This note will describe how to use CMake to build Nginx modules. This hello world nginx module has been used for testing.

Determine Nginx build options

When building a dynamically loadable module one will need the Nginx source code. The Nginx source code used when building the module need to have the same version as the Nginx binary where the module should be loaded.

Check the version and configure arguments using command (output is an example from Nginx in homebrew for OS X):

$ nginx -V
nginx version: nginx/1.23.2
built by clang 14.0.0 (clang-1400.0.29.202)
built with OpenSSL 1.1.1q  5 Jul 2022 (running with OpenSSL 1.1.1s  1 Nov 2022)
TLS SNI support enabled
configure arguments: --prefix=/opt/homebrew/Cellar/nginx/1.23.2 --sbin-path=/opt/homebrew/Cellar/nginx/1.23.2/bin/nginx --with-cc-opt='-I/opt/homebrew/opt/pcre2/include -I/opt/homebrew/opt/openssl@1.1/include' --with-ld-opt='-L/opt/homebrew/opt/pcre2/lib -L/opt/homebrew/opt/openssl@1.1/lib' --conf-path=/opt/homebrew/etc/nginx/nginx.conf --pid-path=/opt/homebrew/var/run/nginx.pid --lock-path=/opt/homebrew/var/run/nginx.lock --http-client-body-temp-path=/opt/homebrew/var/run/nginx/client_body_temp --http-proxy-temp-path=/opt/homebrew/var/run/nginx/proxy_temp --http-fastcgi-temp-path=/opt/homebrew/var/run/nginx/fastcgi_temp --http-uwsgi-temp-path=/opt/homebrew/var/run/nginx/uwsgi_temp --http-scgi-temp-path=/opt/homebrew/var/run/nginx/scgi_temp --http-log-path=/opt/homebrew/var/log/nginx/access.log --error-log-path=/opt/homebrew/var/log/nginx/error.log --with-compat --with-debug --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-ipv6 --with-mail --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module

Make sure that the --with-compat options is there. If not, you may need to use the exact same config options when building the module.

Build a dynamic module

Using a CMake external project is convenient because then source code can be downloaded, configured and built using any command.

Create a CMakeLists.txt file in the module directory and put mandatory directives in the top of the file

cmake_minimum_required(VERSION 3.24)
project(nginx-module)

then create an external project for Nginx like this:

include(ExternalProject)

ExternalProject_Add(
    nginx
    DOWNLOAD_COMMAND URL https://nginx.org/download/nginx-1.23.2.tar.gz
    CONFIGURE_COMMAND cd ${CMAKE_CURRENT_BINARY_DIR}/nginx-prefix/src/nginx && ./configure --add-dynamic-module=${CMAKE_BINARY_DIR}/../ --with-compat
    BUILD_COMMAND make -C ${CMAKE_CURRENT_BINARY_DIR}/nginx-prefix/src/nginx modules
    INSTALL_COMMAND ""
)

As seen the configure command uses --add-dynamic-module to point out the directory of the module. When using CMake a separate subdirectory is used for doing the build in the nginx module directory, that is why we use ${CMAKE_BINARY_DIR}/...

Make sure to use the same version of Nginx as where you will load the module. If not, Nginx will fail to start and show an error when loading the module. This is an example when Nginx 1.23.2 is loading a module build usinh Nginx 1.18 source code:

nginx: [emerg] module "/opt/homebrew/Cellar/nginx/1.23.2/modules/ngx_http_hello_world_module.so" version 1018000 instead of 1023002 in /opt/homebrew/etc/nginx/nginx.conf:11

You can also add a custom command to copy the module shared object into a build output directory to the CMakeLists.txt-file:

add_custom_command(
     TARGET nginx
     COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_SOURCE_DIR}/output
     COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_CURRENT_BINARY_DIR}/nginx-prefix/src/nginx/objs/*.so ${CMAKE_SOURCE_DIR}/output/
     COMMENT "Copy nginx module to ${CMAKE_SOURCE_DIR}/output/"
 )

Then build the module

$ cmake -B build
$ cmake --build build --parallel 8

The module will be found in the output directory.

Load the dynamic module

From the configuration options (seen in nignx -V) you can see from what location Nginx is loading its modules in the option --modules-path. If this option is not specified, the modules are loaded from the modules directory under the path specified in --prefix. In the example above the modules will be loaded from /opt/homebrew/Cellar/nginx/1.23.2/modules

Copy the module to the modules directory. Then in the configuration add

load_module "modules/ngx_hello_world_module.so";

Then start Nginx or reload config

$ nginx -s reload

To test the hello world module just add a test location to the config which is using the directive defined in the module

location /test {
    hello_world;
}

Then reload the config and curl the new location:

$ nginx -s reload
$ curl localhost/test
hello world

Build a static module

Building a static module means that the module is built into the Nginx binary, so you basically build Nginx and specify a path to the module.

Add this CMakeLists.txt into your module directory to build Nginx containing the static module

ExternalProject_Add(
    nginx
    DOWNLOAD_COMMAND URL https://nginx.org/download/nginx-1.18.0.tar.gz
    CONFIGURE_COMMAND cd ${CMAKE_CURRENT_BINARY_DIR}/nginx-prefix/src/nginx && ./configure --add-module=${CMAKE_BINARY_DIR}/../ --with-compat --prefix=/opt/nginx
    BUILD_COMMAND make -C ${CMAKE_CURRENT_BINARY_DIR}/nginx-prefix/src/nginx
    INSTALL_COMMAND make -C ${CMAKE_CURRENT_BINARY_DIR}/nginx-prefix/src/nginx install
)

Build with:

$ cmake -B build
$ cmake --build build --parallel 8

The resulting build can be found in /opt/nginx

To test the hello world module just add a test location using the directive defined in the module. Note that you don’t need load_module since the module is already in the Nginx binary.

location /test {
    hello_world;
}

Then reload the config and curl

$ nginx -s reload
$ curl localhost/test
hello world

References

Nginx Hello World module

Compiling dynamic modules for nginx

Converting static module to dynamic

How dynamic modules work

Set AWS Lambda trigger on an existing S3 bucket in CloudFormation template

Fri, 05 Aug 2022 00:00:00 +0200

Introduction

An AWS Lambda can be triggered by events on an S3 bucket. A very useful feature when e.g. a file upload is made and some operation should be done on the uploaded file. The triggered Lambda can then read the file and perform these operations.

In AWS CloudFormation a Lambda trigger is set on the bucket. When the bucket is defined and deployed in the CloudFormation template a NotificationConfiguration containing a LambdaFunctionConfiguration is set on the bucket. However, setting up such a trigger on an existing bucket (i.e. the bucket exist outside of the stack) is a bit more complex and one method is described here.

Bucket notification configuration using Custom Resource

A solution is to setup the notification configuration on the existing bucket using an AWS CloudFormation Custom resource. An introduction to CloudFormation custom resources can be found in a previous note.

AWS has an article on the subject, however, that description will not allow multiple notification configurations on a bucket since it will overwrite the existing notification configuration (and delete all notification configurations in the delete case).

The solution presented here allows multiple stacks setting bucket notification configuration on the same bucket and has some additional error handling to prevent errors when creating or deleting the stack.

The S3 bucket notification configuration is set as properties on the custom resource. When the lambda resource handler is triggered by the custom resource it reads the notification configuration and adds it to the bucket. When the stack is deleted the lambda will also be triggered and should remove the notification configuration from the bucket. An example is described below.

CloudFormation Template

In this example the existing bucket is passed in as the parameter TestBucket to the stack (the bucket name could also be imported using the intrinsic function Fn::ImportValue). The idea is that a .png file added using the prefix images (e.g. <bucket>/images/faces/me.png) should trigger the Lambda TestLambda provided in the template.

The Custom Resource

  BucketNotifications:
    Type: Custom::S3BucketNotifications
    Properties:
      ServiceToken:
        Fn::GetAtt:
          - BucketNotificationsHandler
          - Arn
      BucketName:
        Ref: TestBucket
      NotificationConfiguration:
        LambdaFunctionConfigurations:
          - Events:
              - s3:ObjectCreated:*
            Filter:
              Key:
                FilterRules:
                  - Name: suffix
                    Value: .png
                  - Name: prefix
                    Value: images
            LambdaFunctionArn:
              Fn::GetAtt:
                - TestLambda
                - Arn
      Managed: false
    DependsOn:
      - AllowTestBucketInvokeTestLambda

The ServiceToken property is specifying the Lambda resource handler that will be invoked when the Custom Resource is created. The rest of the properties are to be read by that Lambda resource handler.

The NotificationConfiguration is the actual configuration to be set on the bucket. Other than that there is the Managed and BucketName properties, the latter is obvious but Managed might need a clarification. Managed set to true means that the bucket is created in this template. That’s not the case here, hence Managed is set to false.

When the bucket exist outside of the stack the bucket notification handler need to be cautious that there could be other notifications on the bucket. Those notifications should not be affected by the bucket notification handler. In the other case, when the bucket is Managed (i.e. created in the stack) no such considerations need to be made.

Custom Resource handler

This is the actual bucket notification handler. A lambda with inline python code. The code is actually what AWS CDK is generating when setting bucket notification configuration using the CDK.

  BucketNotificationsHandler:
    Type: AWS::Lambda::Function
    Properties:
      Description: AWS CloudFormation handler for "Custom::S3BucketNotifications" resources (@aws-cdk/aws-s3)
      Handler: index.handler
      Role:
        Fn::GetAtt:
          - BucketNotificationsHandlerRole
          - Arn
      Runtime: python3.9
      Timeout: 300
      Code:
        ZipFile: |
          import boto3  # type: ignore
          import json
          import logging
          import urllib.request

          s3 = boto3.client("s3")
          l = boto3.client("lambda")

          CONFIGURATION_TYPES = ["TopicConfigurations", "QueueConfigurations", "LambdaFunctionConfigurations"]

          def handler(event: dict, context):
              response_status = "SUCCESS"
              error_message = ""
              try:
                  props = event["ResourceProperties"]
                  bucket = props["BucketName"]
                  notification_configuration = props["NotificationConfiguration"]
                  request_type = event["RequestType"]
                  managed = props.get('Managed', 'true').lower() == 'true'
                  stack_id = event['StackId']

                  if managed:
                    config = handle_managed(request_type, notification_configuration)
                  else:
                    config = handle_unmanaged(bucket, stack_id, request_type, notification_configuration)

                  put_bucket_notification_configuration(bucket, config)
              except Exception as e:
                  logging.exception("Failed to put bucket notification configuration")
                  response_status = "FAILED"
                  error_message = f"Error: {str(e)}. "
              finally:
                  submit_response(event, context, response_status, error_message)


          def handle_managed(request_type, notification_configuration):
            if request_type == 'Delete':
              return {}
            return notification_configuration


          def handle_unmanaged(bucket, stack_id, request_type, notification_configuration):

            # find external notifications
            external_notifications = find_external_notifications(bucket, stack_id)

            # if delete, that's all we need
            if request_type == 'Delete':
              return external_notifications

            def with_id(notification):
              notification['Id'] = f"{stack_id}-{hash(json.dumps(notification, sort_keys=True))}"
              return notification

            # otherwise, merge external with incoming config and augment with id
            notifications = {}
            for t in CONFIGURATION_TYPES:
              external = external_notifications.get(t, [])
              incoming = [with_id(n) for n in notification_configuration.get(t, [])]
              notifications[t] = external + incoming
            return notifications


          def find_external_notifications(bucket, stack_id):
            existing_notifications = get_bucket_notification_configuration(bucket)
            external_notifications = {}
            for t in CONFIGURATION_TYPES:
              # if the notification was created by us, we know what id to expect
              # so we can filter by it.
              external_notifications[t] = [n for n in existing_notifications
                .get(t, []) if not n['Id'].startswith(f"{stack_id}-") and lambda_exist(n['LambdaFunctionArn'])]

            return external_notifications


          def lambda_exist(lambda_arn):
            try:
              l.get_function(FunctionName=lambda_arn)
            except Exception as e:
              if 'ResourceNotFound' in str(e):
                print(f'lambda {lambda_arn} does not exist. {e}')
                return False
            return True


          def get_bucket_notification_configuration(bucket):
            return s3.get_bucket_notification_configuration(Bucket=bucket)


          def put_bucket_notification_configuration(bucket, notification_configuration):
            s3.put_bucket_notification_configuration(Bucket=bucket, NotificationConfiguration=notification_configuration)


          def submit_response(event: dict, context, response_status: str, error_message: str):
              response_body = json.dumps(
                  {
                      "Status": response_status,
                      "Reason": f"{error_message}See the details in CloudWatch Log Stream: {context.log_stream_name}",
                      "PhysicalResourceId": event.get("PhysicalResourceId") or event["LogicalResourceId"],
                      "StackId": event["StackId"],
                      "RequestId": event["RequestId"],
                      "LogicalResourceId": event["LogicalResourceId"],
                      "NoEcho": False,
                  }
              ).encode("utf-8")
              headers = {"content-type": "", "content-length": str(len(response_body))}
              try:
                  req = urllib.request.Request(url=event["ResponseURL"], headers=headers, data=response_body, method="PUT")
                  with urllib.request.urlopen(req) as response:
                      print(response.read().decode("utf-8"))
                  print("Status code: " + response.reason)
              except Exception as e:
                  print("send(..) failed executing request.urlopen(..): " + str(e))
    DependsOn:
      - BucketNotificationsHandlerRoleDefaultPolicy
      - BucketNotificationsHandlerRole

Basic idea

The Lambda will find all notifications on the bucket and will then add or remove the notification configuration for the current stack to/from the list of notification configurations and then re-set the configuration on the bucket.

Handle non-existing lambda trigger destination

As mentioned the custom resource lambda handler code is what CDK is generating - with one addition. There is a check added to make sure the lambda that should be triggered exists (the call to lambda_exist(). Otherwise, if the lambda does not exist any operation (create, update or delete) on the stack would fail. This is a corner case but can happen when having multiple CloudFormation stacks where the stacks contain a lambda that is triggered by an external S3 bucket and the stacks are deleted in parallel.

The call Traceback for the error would look like;

Failed to put bucket notification configuration
Traceback (most recent call last):
File "/var/task/index.py", line 27, in handler
put_bucket_notification_configuration(bucket, config)
File "/var/task/index.py", line 87, in put_bucket_notification_configuration
s3.put_bucket_notification_configuration(Bucket=bucket, NotificationConfiguration=notification_configuration)
File "/var/runtime/botocore/client.py", line 391, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 719, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Unable to validate the following destination configurations

I.e. the destination notification configuration is incorrect since one of the lambdas does not exist.

Permissions

The S3 bucket need to have InvokeFunction permission on the target Lambda

  AllowTestBucketInvokeTestLambda:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName:
        Fn::GetAtt:
          - TestLambda
          - Arn
      Principal: s3.amazonaws.com
      SourceAccount:
        Ref: AWS::AccountId
      SourceArn:
        !Join
          - ''
          - - 'arn:aws:s3:::'
            - !Ref TestBucket

The custom resource handler lambda should assume Lambda Basic Execution Role (for logging to CloudWatch Logs):

  BucketNotificationsHandlerRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
        Version: "2012-10-17"
      ManagedPolicyArns:
        - Fn::Join:
            - ""
            - - "arn:"
              - Ref: AWS::Partition
              - :iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

The custom resource handler lambda need permission to add bucket notification on the bucket

  BucketNotificationsHandlerRoleDefaultPolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyDocument:
        Statement:
          - Action: s3:*BucketNotification
            Effect: Allow
            Resource: "*"
        Version: "2012-10-17"
      PolicyName: BucketNotificationsHandlerRoleDefaultPolicy
      Roles:
        - Ref: BucketNotificationsHandlerRole

Summary

A working example can be found here.

Deploy the template using AWS Console or AWS Cli using:

$ aws cloudformation create-stack --stack-name bucket-notification-demo --template-body file://bucketnotification.yaml --capabilities CAPABILITY_IAM --parameters ParameterKey=TestBucket,ParameterValue=<your-bucket>

Resources

AWS Custom Resource

AWS Notification config on existing bucket with cloudformation

Introduction to AWS CloudFormation Custom Resource

Thu, 31 Mar 2022 00:00:00 +0200

Introduction

This note describes the AWS CloudFormation Custom resource.

A custom resource that fetches the API Key from a deployed AWS API Gateway is shown as an example to demonstrate how custom resources work and can be used.

CloudFormation Custom Resource

A Custom Resource has only one required property, the ServiceToken. The ServiceToken can either be the ARN of a Lamda or an SNS. In the case of a lambda the lambda will be called on create, delete or update of the custom resource. When the lambda is called all properties of the custom resource can be found in the ResourceProperty of the request object.

A custom resource need to signal that deployment succeeded (or failed) and does so by sending a response object to the response URL that can be found in the ResponseURL of the request object.

This is basically how a Custom Resource is defined in the template. In this example, a CloudFormation parameter is passed in to the Custom Resource using Fn::Ref Intrinsic function and the parameter should contain the API Key Id.

Parameters:
  ApiKeyIdParam:
    Type: String

Resources:
  GetApiKey:
    Type: Custom::GetApiKey
    Properties:
      ServiceToken: !GetAtt GetApiKeyResourceHandler.Arn
      ApiKeyId: !Ref ApiKeyIdParam

The Arn for the Lambda to be invoked is assigned to the ServiceToken property, in this example the Lambda resource is called GetApiKeyResourceHandler. The parameter ApiKeyId passed in as a property will be read by the handler lambda GetApiKeyResourceHandler.

Resource handler Lambda

Any input needed by the logic implemented in the Custom Resource should be set as properties on the custom resource. The properties will then be available in the ResourceProperty of the request object and can be read by the lambda handler. The lambda handler can then e.g. use AWS SDK to read or manipulate other resources.

  GetApiKeyResourceHandler:
    Type: AWS::Lambda::Function
    Properties:
      Description: Handler for GetApiKey Custom Resource
      Handler: index.handler
      Runtime: python3.9
      Role: !GetAtt GetApiKeyResourceHandlerRole.Arn
      Code:
        ZipFile: |
          import boto3
          import json
          import urllib.request

          def handler(event, context):
              response_status = "SUCCESS"

              if event['RequestType'] != 'Create':
                # Only care about stack Create opertations
                send_response(event, response_status)
                return

              api_key_id = event['ResourceProperties'].get('ApiKeyId')
              client = boto3.client('apigateway')
              message = ''
              data = {}

              if api_key_id:
                  try:
                      r = client.get_api_key(apiKey = api_key_id, includeValue=True)
                      # Add output
                      data = { "ApiKey": r['value'] }
                  except Exception as e:
                      response_status = "FAILED"
                      print(f'Error: {e}')
                      message = f'Failed to deploy {str(e)}'

              send_response(event, response_status, message, data)

          def send_response(event, status, message='', data={}):
              # Fill the required response properties
              response = {
                  "Status": status,
                  "Reason": message,
                  "PhysicalResourceId": event.get('PhysicalResourceId') or event['LogicalResourceId'],
                  "StackId": event['StackId'],
                  "RequestId": event['RequestId'],
                  "LogicalResourceId": event['LogicalResourceId'],
                  "Data": data}
              # Send the response
              response_body = json.dumps(response).encode('utf-8')
              req = urllib.request.Request(url=event['ResponseURL'], data=response_body, method="PUT")
              with urllib.request.urlopen(req) as r:
                  print(r.read().decode('utf-8'))

The GetApiKeyResourceHandler is a Lambda with inline python code. As defined in the template the handler is the function named handler() and starts by reading the property that was passed in. Then the Lambda is getting the API key from the API Gateway using the AWS Python SDK (called Boto3),

Finally a response is sent to the ResponseURL that can be found in the request object passed to the lambda. The response object should have Status: SUCCESS to signal that the resource was successfull deployed.

Request Type

The request object also contains a RequestType attribute which makes it possible to check if the custom resource is created, updated or deleted. Available RequestTypes are Create, Update, Delete.

Error handling

If there are errors in the logic, the Status attribute in the response object should be set to Status: FAILED. Setting the status to FAILED will cause the deploy to fail and the CloudFormation stack will rollback. You can also set a helpful error message in the Reason property: "Reason": "Failed to ....". The error message will be shown in the CloudFormation event log.

Return value from a Custom Resource

The Custom Resource Handler may produce a value or some other result that need to be propagated further. This is done by setting a Data object in the response object. In the example above the Data object is set to:

data = { "ApiKey": r['value'] }

And then that value can be passed as an output (or passed to other resources) on the CloudFormation stack using the GetAtt intrinsic function.

Outputs:
  ApiKey:
    Description: 'API Key as Output from Custom Resource'
    Value: !GetAtt GetApiKey.ApiKey

Permissions

Define the role referenced by the Custom Resource Handler. This role need to have policies that allows the lambda to perform its logic. In this example the logic need permission to perform apigateway:GET action. Also the AWSLambdaBasicExecutionRole is set (which is only a policy allowing to write CloudWatch logs).

  GetApiKeyResourceHandlerRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
        Version: "2012-10-17"
      ManagedPolicyArns:
        - Fn::Join:
          - ""
          - - "arn:"
            - Ref: AWS::Partition
            - :iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: AllowApiGatewayGet
          PolicyDocument:
            Statement:
            - Action: apigateway:GET
              Effect: Allow
              Resource: "*"
            Version: "2012-10-17"

Problems and Caveats

It is possible to screw up a bit. Any errors in the Custom Resource handler code which makes the lambda crash before the response is sent will make CloudFormation wait until there is a timeout. The timeout is very long, like 30 minutes or so.

Note that this also happens on errors that you normally don’t handle in the code with exceptions e.g. syntax errors or importing unavailable modules.

The stack will stay in CREATE_IN_PROGRESS or if the error happens when the stack is deleted it will stay in DELETE_IN_PROGRESS until CloudFormation times out.

The problem causing the crash can probably be seen in the CloudWatch logs for the Custom Resource handler. In this example they are found in `CloudWatch/Log Groups/aws/lambda/-GetApiKeyResourceHandler

Summary

The CloudFormation template used in this example can be found on GitHub.

Deploy the template using AWS Console or using AWS CLI:

$ aws cloudformation create-stack --stack-name custom-resource-demo --template-body file://template.yaml --capabilities CAPABILITY_IAM --parameters ParameterKey=ApiKeyIdParam,ParameterValue=xxxyyyzzza

Resources

Custom Resource request

Custom Resource response

Hide the Android navigation bar using WindowInsetsController

Wed, 15 Sep 2021 00:00:00 +0200

Introduction

Documentation describing how to hide the navigation bar and/or running an Android application in fullscreen usually uses the View.SYSTEM_UI_FLAG_HIDE_NAVIGATION and View.SYSTEM_UI_FLAG_FULLSCREEN flags, e.g. Android developer documents and a another Android developer doc.

However SYSTEM_UI_FLAG_FULLSCREEN and SYSTEM_UI_FLAG_HIDE_NAVIGATION are marked deprecated together with the other SYSTEM_UI-flags. This note describes how to hide navigation bar in the new way using the WindowInsetsController.

Example

There is a working demo/test written in Kotlin of WindowInsetsController on my github

Fullscreen mode

The WindowInsetController class have methods that is supposed to replace the use of the deprecated SYSTEM_UI_FLAG_FULLSCREEN, however the methods used for hiding the bars are only available in Android 11 and newer (i.e. API Level > 30). This is solved by using the wrapper class WindowInsetsControllerCompat who will handle checking the API-level of the device running the application and make it work for all API-levels.

For fullscreen, hide the navigation and status bars.

WindowInsetsControllerCompat controller = new WindowInsetsControllerCompat(getWindow(), getWindow().getDecorView());
controller.hide(WindowInsetsCompat.Type.systemBars());

To allow the bars to be shown when swiping from the edge of the screen one can set system bars behaviour like this:

controller.setSystemBarsBehavior(WindowInsetsControllerCompat.BEHAVIOR_SHOW_TRANSIENT_BARS_BY_SWIPE);

Under some circumstances the navigation bar did not show up when swiping from the edge of the screen on my application. Unsure why it happened but it was often the case after the screen was rotated.

I made a workaround to show the navigation bar when tapping on the screen instead.

Show the navigation bar for 3s when tapping on the layout:

mainlayout.setOnClickListener(new View.OnClickListener() {
    @Override
    public void onClick(View v) {
        View decorView = getWindow().getDecorView();
        WindowInsetsControllerCompat controller = new WindowInsetsControllerCompat(getWindow(), decorView);
        controller.show(WindowInsetsCompat.Type.navigationBars());
        decorView.postDelayed(new Runnable() {
            @Override
            public void run() {
                controller.hide(WindowInsetsCompat.Type.systemBars());
            }
        }, 3000);
    }
});

Note that the above is an example and it may not be possible to use the main layout for capturing the click events.

When the Navigation bar is auto-hiding it looks nice if it also is made translucent. Add the following to the style element in themes.xml (or the resource file containing the style e.g. style.xml)

    <item name="android:windowTranslucentNavigation">true</item>
    <item name="android:fitsSystemWindows">true</item>

The second line is needed in order to make the activity content display correct. Without it some parts of the content may be laid out outside of the screen like this:

Compared to when using a translucent navigation bar and fitsSystemWindows: true:

Resources

Exploring WindowInsets on Android 11

Setup TCP and UDP forwarding to Android Emulator

Wed, 25 Aug 2021 00:00:00 +0200

Introduction

This is a note describing the steps needed to forward TCP and UDP traffic to the Android Emulator.

Sometimes when developing an application one need to connect and send data from the host machine to the app running in an Android Emulator. In order to do that there are some tricks needed for forwarding the traffic.

Forward TCP traffic to Android Emulator

Forwarding TCP traffic to android emulator is simple. Just use the adb tool like this to forward host port 2222 to port 3333 on the Android emulator:

$ adb forward tcp:2222 tcp:3333

Check with

$ adb forward --list
emulator-5554 tcp:2222 tcp:3333

Forward UDP traffic to Android Emulator

The forward command in adb tool can only forward TCP traffic. In order to forward UDP traffic one need to telnet into the emulator itself and run a command.

First you need the emulators auth token. The auth token can be found by doing

$ cat ~/.emulator_console_auth_token

When the emulator is running it opens a control port on port 5554, Connect to the port using telnet and use the command

auth <token>

to authenticate. Then use the redir command to forward host UDP port 2222 to emulator port 3333. This is the complete procedure:

$ telnet localhost 5554
Trying ::1...
Connected to localhost.
Escape character is '^]'.
Android Console: Authentication required
Android Console: type 'auth <auth_token>' to authenticate
Android Console: you can find your <auth_token> in
'/Users/bjorn/.emulator_console_auth_token'
OK
auth sU6**********XS8
Android Console: type 'help' for a list of commands
OK
redir
allows you to add, list and remove UDP and/or PORT redirection from the host to the device
as an example, 'redir  tcp:5000:6000' will route any packet sent to the host's TCP port 5000
to TCP port 6000 of the emulated device

available sub-commands:
    list             list current redirections
    add              add new redirection
    del              remove existing redirection

redir add udp:2222:3333
OK
redir list
udp:2222  => 3333
OK

So as seen in the redir list command host UDP port 2222 is now forwarded to port 3333 on the emulator.

Resource

Android Developer - emulator networking

Setting up software RAID in linux

Tue, 21 Jul 2020 00:00:00 +0200

Introduction

Setting up software RAID in Linux is very easy, it is also possible to simulate a disk failure and verify that the array is working. The installation described here was made on ArchLinux but apart from installation command the instructions should be the same for any Linux distribution.

Installing Software RAID manangement tool

Install the linux md tool in ArchLinux:

$ sudo pacman -S mdadm

Configuring RAID array

I have four disks sdb, sdc, sdd, and sde which I want to add to a RAID5 array.

$ cat /proc/partitions 
major minor  #blocks  name

   8        0    8388608 sda
   8        1    8387584 sda1
   8       16     524288 sdb
   8       32     524288 sdc
   8       48     524288 sdd
   8       64     524288 sde

In this example my disks are four 512MB disks (this example is actually from a virtual machine). RAID5 means that size for the array will be the sum of all minus 1 of the disks. I.e. in this case with four 0.5GB disks the size of the array will be 1.5GB. The non available space is used to store checksums for data which makes it possible to recreate data from a failed disk. In RAID5 it is possible to recover from one failed disk. RAID6 can recover from two failed disks, but RAID6 will also use 2 disks that are unavailable for storage on the array.

Creating a RAID5 array

$ sudo mdadm --create /dev/md0 --chunk=32K --level=raid5 --raid-devices=4  /dev/sdb /dev/sdc /dev/sdd /dev/sde
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

Use the device

Create filesystem and mount

$ sudo mkfs.ext4 /dev/md0
$ sudo mount /dev/md0 /media/raid

Checking the raid array progress

Check the status of the RAID array:

$ cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      1569792 blocks super 1.2 level 5, 32k chunk, algorithm 2 [4/4] [UUUU]
      
unused devices: <none>

[UUUU] shows the status of each disk in the array, all disks are UP. A degraded array would show _ for the faulty drive.

To get more details on the raid array:

$ sudo mdadm -D /dev/md0 
/dev/md0:
           Version : 1.2
     Creation Time : Tue May 19 22:19:54 2020
        Raid Level : raid5
        Array Size : 1569792 (1533.00 MiB 1607.47 MB)
     Used Dev Size : 523264 (511.00 MiB 535.82 MB)
      Raid Devices : 4
     Total Devices : 4
       Persistence : Superblock is persistent

       Update Time : Tue May 19 22:20:01 2020
             State : clean 
    Active Devices : 4
   Working Devices : 4
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 32K

Consistency Policy : resync

              Name : arch:0  (local to host arch)
              UUID : 35300d92:56f766ae:d945154a:daa7456e
            Events : 18

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       2       8       48        2      active sync   /dev/sdd
       4       8       64        3      active sync   /dev/sde

Testing the RAID array

Simulate a faulty drive:

$ sudo mdadm --manage --set-faulty /dev/md0 /dev/sdb
mdadm: set /dev/sdb faulty in /dev/md0

The drive /dev/sdb is now faulty but the array can still be used.

To recover the faulty drive remove the drive from the raid

$ sudo mdadm /dev/md0 -r /dev/sdb
mdadm: hot removed /dev/sdb from /dev/md0

Add drive to the raid array again

$ sudo mdadm /dev/md0 -a /dev/sdb
mdadm: added /dev/sdb

The array will now start to rebuild. The rebuild process can be monitored on /proc/mdstat

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb[6] sde[4] sdd[5] sdc[1]
      1569792 blocks super 1.2 level 5, 32k chunk, algorithm 2 [4/3] [_UUU]
      [========>............]  recovery = 44.5% (233692/523264) finish=0.0min speed=233692K/sec

After the progress bar there is an estimate on how much time is left for the rebuild process, in this case the disks are so small so the rebuild process is really fast. If the disks are large and the system is in use with some disk IO load and CPU load the process can take quite a lot of time.

To see what is happening one can monitor disk io and cpu load during rebuild


$ iostat -k 5

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,21    0,00   49,06    2,49    0,00   48,23

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
md0               0,00         0,00         0,00         0,00          0          0          0
sda               0,40         0,00         3,20         0,00          0         16          0
sdb             142,80        30,70     84257,50         0,00        153     421287          0
sdc             137,20     84793,60         1,40         0,00     423968          7          0
sdd             137,00     84792,80         1,40         0,00     423964          7          0
sde             137,00     84792,80         1,40         0,00     423964          7          0
scd0              0,00         0,00         0,00         0,00          0          0          0

As can be seen on the disk IO, data is read from sdc/sdd/sde and written to sdb. This system was completly unused during the rebuild process.

References

RAID setup

Explaining mdstat info

Speeding up RAID rebuild time

Support Google AdMob Ads in Godot Android games

Sat, 16 May 2020 00:00:00 +0200

Introduction

In a previous note I wrote about creating Android games from an existing Godot project . This is a continuation on that note and will explore the possibility to add Google AdMob ads in an Android game created with Godot. Adding AdMob ads support to Godot requires rebuilding the export templates, hence, that step is described in detail here.

Google AdMob

When using AdMob for real you need to create an application and an Ad Unit in Google AdMob. It is also possible to use test ads when developing the game.

Create a Google AdMob account. Then sign in and create an application by clicking Apps –> Add App. You will be asked if you have uploaded the App to Play Store. Next you need to setup the new app, write a name and select platform:

Now your app will get an AdMob AppId which is needed later when exporting the game from Godot. Click on Create Ad Unit

Select an Ad format, in this example a Banner used.

In the next step add a name for the ad unit. You can make some additional configuration in the Advanced Setting, e.g. choose if the ad should play a movie etc. When finished click Create Ad Unit

Now you have an Ad Unit Id and an App Id. Both will be needed later.

You can later find your App and Ad Unit in AdMob user interface where you also will be able to find the App Id and Unit Id.

Build export templates with support for third party AdMob module

There are a few cases when rebuilding export templates are required. In the Android case for example any changes that normally are done in a projects build.gradle like targetSdkVersion and minSdkVersion require you to rebuild the export templates.

Adding a third party module in Godot may also require you to rebuild the export templates. In this case the AdMob module for supporting ads in the exported Android game is being added.

Getting Godot and the AdMob module

To build the export templates you need to clone the godot source code.

Clone godot source from github and checkout the same version tag as the Godot version you are running, in my case I am running ArchLinux and currently I have Godot version 3.1.2 installed i.e:

$ git clone https://github.com/godotengine/godot.git
$ cd godot
$ git checkout 3.1.2-stable
$ cd ..

Clone the AdMod github repo https://github.com/kloder-games/godot-admob

$ git clone https://github.com/kloder-games/godot-admob.git

then copy the admob directory to godot/modules directory.

$ cp -r godot-admob/admob godot/modules/

Building

Install Dependencies

Godot is using the scons build system. Java-8 is needed to build Android export templates.

$ sudo pacman -S jdk8-openjdk scons

Android SDK and Android NDK are needed, download and unpack them. Android SDK is installed by installing Android Studio. Then set the environment variables ANDROID_HOME and ANDROID_NDK_ROOT to where they are located

$ export ANDROID_HOME=/home/bjorn/android-sdk/
$ export ANDROID_NDK_ROOT=/home/bjorn/android-ndk-r21b/

Build Godot Libraries for Android

It should now be possible to start building the Godot libraries. This will take a while. You need to specify platform, target and architecture.

First specify the architectures to build for. Google Play Store requires that 64-bit architectures are supported so build at least the architectures armv7 and arm64v8. If the application should be running in an emulator, make sure to build x86 (in case the emulator is using that architecture, which is recommeded for performace reasons).

If you want to load the application on an Android phone without Play Store it is easier if you also build debug target (then you don’t need to sign with your own key, it is enough with the default debug key provided with Android SDK), but then debug versions of the Godot libraries need to be built.

$ cd godot
$ scons platform=android target=release android_arch=armv7
$ scons platform=android target=release android_arch=arm64v8
$ scons platform=android target=release android_arch=x86
$ scons platform=android target=debug android_arch=armv7
$ scons platform=android target=debug android_arch=arm64v8
$ scons platform=android target=debug android_arch=x86

When all builds are completed there should be 2 library files for each build, libc++_shared.so and libgodot_android.so, in godot/platform/android/java/libs/<target>/<arch>

If, for some reason, the builds need to be redone clean with:

$ scons platform=android --clean

Update Android files

Building libraries will also create AndroidManifest.xml and build.gradle in platform/android/java. Those files need some changes before building export templates.

You need to add the application id for your app in Google AdMod in AndroidManifest.xml. Open godot/platform/android/java/Manifest.xml:

Replace:

<meta-data
    android:name="com.google.android.gms.ads.AD_MANAGER_APP"
    android:value="true"/>

With:

<meta-data
    android:name="com.google.android.gms.ads.APPLICATION_ID"
    android:value="<your-application-id>"/>

Where <your-application-id> is replaced with the Application id from Google AdMob see Google AdMob section.

In my case I also want to change minSdkVersion to only support a bit newer Android versions. Default this is set to 18 which is Android version 4.3 which is from 2013. I update this to 22 which is Android 5.1 from 2015. Open godot/platform/android/java/build.gradle and set

minSdkVersion 22

Note that if you rebuild the Android Godot libraries the files you changed here will be overwritten and hence you need to update them again.

This also means that you need to build separate export templates for every application you have, since the Application Id is specified in AndroidManifest.xml. This is unfortunate but I haven’t yet found any other way.

Upgrade AdMob SDK

This is an optional step and can be skipped.

Current version of AdMob module uses AdMod SDK version 16.0.0. The latest version is currently 19.1.0 so let’s try to use that. Open build.gradle and replace

compile ('com.google.android.gms:play-services-ads:16.0.0') { exclude group: 'com.android.support' }

with

compile ('com.google.android.gms:play-services-ads:19.1.0') { exclude group: 'androidx.core' }

Then you also need to add the following to packagingOptions

exclude 'META-INF/androidx.legacy_legacy-support-core-utils.version'
exclude 'META-INF/androidx.loader_loader.version'
exclude 'META-INF/androidx.localbroadcastmanager_localbroadcastmanager.version'
exclude 'META-INF/androidx.print_print.version'
exclude 'META-INF/androidx.documentfile_documentfile.version'

Build Android export templates

Now it’s time to build the export template apk's

$ cd platform/android/java/
$ ./gradlew build

The result from this build step are the files godot/bin/android_release.apk and godot/bin/android_debug.apk.

The export template build can be cleaned with

$ ./gradlew clean

Troubleshooting build problems

In ArchLinux the build may fail with

* What went wrong:
Could not open terminal for stdout: could not get termcap entry

This is solved by setting the TERM environment like this:

$ export TERM=xterm-color

Adding Google Ads to the Android game

Time to add some code to the Godot project for interacting with AdMob SDK.

Showing Ads in Godot

Showing the ads is simple. First some initialization and then load the ad. This is an example for a banner:

var admob = null
if (Engine.has_singleton("AdMob")):
    admob = Engine.get_singleton("AdMob")
    admob.init(false, get_instance_id())
    admob.loadBanner("ca-app-pub-3940256099942544/6300978111", false)

Then you can show and hide the banner as you wish with:

admob.showBanner()
admob.hideBanner()

Note that the code above will show test ads. The banner ad unit id is a test banner from Google, make sure to replace it with the real ad unit id (that was created in Google AdMob) and also change to admob.init(true, get_instance_id()) because the first argument is whether the ad is real or test (true means it is a real ad).

Exporting the game

For instruction on how to export the game follow the procedure, with the modifications mentioned below, described in Create Android Game Applications from Godot.

In addition to the linked description: In Custom package select the export template apk’s you just built both for debug and release.

Make sure to enable Internet and Access Network State in privilegies since the AdMob module need them.

For exporting to Play Store make sure to uncheck the Export with Debug checkbox.

Example

This is an example of a banner ad in the Godot sample game Platformer. The banner is only shown on the game menu and not when playing the game, hence, the banner can cover the game navigation buttons.

There is also an interstitial ad when exiting the game.

You can install and try the sample game from Google Play Store and you can also find the source code on GitHub.

References

Compile for Android

Export to Android

Create Android Game Applications from Godot

Mon, 11 May 2020 00:00:00 +0200

Introduction

Godot is an open source game development framework that can be used to create games for desktop or mobile devices. This note describe how to create applications for Linux and Android from an existing Godot project.

Setting up Godot for creating native application

A project in Godot can be “exported” to applications for different platforms. To be able to export the application an export template for the target platform is needed.

Installing export templates

In Godot menu go to Editor -> Manage Export Templates:

In the dialog Export template manager click Download to install export templates for the current version, you then need to select a mirror to download from in the next dialog. The export templates are around 400MB to download and are installed in $HOME/.local/share/godot/templates/<version>/.

Setting up Android export template

Android SDK is needed for creating an Android application so setting up Godot for exporting to Android require some paths to binaries that comes with Android SDK.

In Godot menu go to Editor -> Editor Settings and scroll down to Export/Android and add path to adb and jarsigner. adb comes with Android SDK, jarsigner comes with Java SDK and in my case they are installed in /usr/bin/adb and /usr/bin/jarsigner. Then add path to the debug keystore, if you have worked with Android Studio and created an application, you should already have a debug keystore in /home/<user>/.android/debug.keystore, add the keystore user androiddebugkey and keystore password android.

Creating Android applications with Godot

For installation on local device

When installing an application on the device you have in hand you can manage with the simplest way of export, Godot will sign the app with the debug key so you don’t need to create a signing key.

Open the Godot project that should be exported to an android app.

In Godot menu go to Project -> Export click Add… and select Android.

In Export Path enter a Package Name including path to where the .apk should be exported. If none is given the default is that the project directory is opened in the export step.

Under package, write a package name and check the singed checkbox.

Select a launcher icon, if there are any (otherwise there will be a default Godot icon for the app).

Check the architectures

Then you need to check Permissions needed for the application, like e.g. Access Network State or Internet in case your app need any persmissions.

Click the button Export Project

Ensure that the path is correct and select a feasible name for the apk-file.

Check “Export With Debug”

Click “Save”

The resulting apk-file can be installed on an Android device with (see my note on how to install an Android application apk-file):

$ adb install <apk-file>

However, it is not possible to upload this apk to Google Play Store since it contains debug info and is only signed with a debug key.

For publishing on Google Play Store

There are a couple of things needed to upload the application to Google Play Store.

Google Play Store requires 64-bit version of the application so make sure that Arm 64-v8a architecture is checked.

The other thing is that you need to sign the application with a real signing key, in Google Play Console this is called Upload Key. Also the app cannot contain debug info so you need to use a release build.

Create a release keystore like this:

$ keytool -v -genkey -v -keystore bhj.keystore -alias bhj-game -keyalg RSA -validity 10000

The tool will ask for keystore password, then some certificate information and finally asking you to enter a password for the key, in Godot (at least in version 3.1.2) one have to set the same password for the key and for the keystore.

Save this keystore file in a safe place and remember the password you used for the key and keystore. Any updates you will do to this application need to be signed with the same key.

You can list the keystore with

$ keytool -list --keystore bhj.keystore

Then when exporting the application you should now scroll down to KeyStore and add the keystore file (including path) to Release the key alias to Release User and the password (which is the same for the keystore and for the key) to Release password.

You should then be able to export the application using the procedure described above with one exception. You need to uncheck Export with Debug before clicking the Save button since debug information is not allowed on apps in Play Store for security reasons.

The resulting apk-file should be possible to upload to Google Play Store using Google Play Console. Read more about how to publish an Android application on Google Play Store in my earlier post.

Tweaking Android details

Developers familiar with Android applications may wonder about Target SDK and other things that can be specified in gradle build files. Unfortunately this will require you to rebuild the export templates. For example changing minSdkVersion to a higher Android version will require a rebuild of the export templates. Update: More on building export template in this later note.

Linux, Windows and OSX

There are templates for exporting to a number of other platforms as well, of which I have tested only exporting to Linux. Export templates for Linux and the other target are installed when doing the step Installing export templates above.

Go to Project -> Export click Add... in the dialog and select a target to export to:

The procedure is simpler when exporting applications to desktops (Linux, Windows and OSX) than for mobile targets Android and iOS.

References

Godot Docs - Export to Android

Installing an Android application (.apk) file

Sat, 22 Feb 2020 00:00:00 +0100

Introduction

Installing an Android application package (an apk-file) when developing an android application should be a rather trivial task. However, there are a few tweaks needed to make it work in Linux. This note describes how you install Android apk-files on a device from a Linux host. I have followed this procedure on ArchLinux and on Ubuntu.

In order to proceed you need Android SDK installed on the Linux host.

Turn on Developer mode on your Android phone

You need to turn on developer mode and allow usb-debugging. The way this is done may depend on the Android device used.

On Motorola Moto e6 (running Android 9) you go to settings, then System, advanced -> About phone then hit Build number multiple times.

When developer mode is turned on there should be a new menu option in Settings -> system -> Developer options where you can scroll down to Debugging and enable USB Debugging.

No permissions for device

When you plug your device you can use adb (available from Android SDK) to list devices.

$ adb devices -l
List of devices attached
ZE2222MG7R             no permissions; see [http://developer.android.com/tools/device.html] usb:2-2 transport_id:1

as seen there are no permission to install an application via adb on the connected device. This can be solved by adding a udev rule.

USB device id

To create the udev rule the the USB Device id is needed so let’s list the device id using lsusb (this can also be found in dmesg when the device is connected)

$ lsusb
Bus 002 Device 006: ID 0e8d:201c MediaTek Inc. moto e6 play

Here the vendor id is 0e8d and the product id is 201c.

Adding udev rule

Create a rules file for Android in /etc/udev/rules.d/51-android.rules and add the line

SUBSYSTEM=="usb", ATTR{idVendor}=="0e8d", ATTR{idProduct}=="201c", MODE="0660", OWNER="bjorn"

Now reload udev

$ sudo udevadm control --reload

Disconnect and reconnect the device.

Now list devices again

$ adb devices -l
List of devices attached
ZE2222MG7R             unauthorized usb:2-2 transport_id:6

There should be a pop-up dialog on the device asking you to allow USB debugging.

Authorize and list devices again

$ adb devices -l
List of devices attached
ZE2222MG7R             device usb:2-2 product:bali_reteu model:moto_e6_play device:bali transport_id:7

Now you should be allowed to install applications on the device using adb

Installing the app

Now you can install the app using Android Studio or via adb. In adb you will probably need to allow installing test packages, so add the -t flag.

$ adb install -t <package>

If you already have the package installed and want to replace the existing, add the -r flag

$ adb install -t -r <package>

Writing a remote shell in Java

Sun, 19 Jan 2020 00:00:00 +0100

This note describes how to write a simple remote shell in Java.

Remote shell

A remote shell application obviously need to be able to run a command on the host computer, it also need some kind of socket server in order for clients to run commands remotely.

Executing command

The application need to execute a command and grab its output. The output should then be sent to the remote client.

Process command = Runtime.getRuntime().exec(commandStr);
BufferedReader reader = new BufferedReader(
                        new InputStreamReader(command.getInputStream()));

So this code will run the command in commandStr and then get the InputStream from the Process object (where input means input to the application, i.e. the output from the command). A reader is created for the stream and will later be used to grab the output from the command.

Reading command output

To read the command output just use the reader created earlier:

String line = "";
while ((line = reader.readLine()) != null) {
    System.out.println(line);
}
command.waitFor();

The loop will read all outputs from the command and terminates on end of line. The command.waitFor() waits for the command to exit.

Starting the server

Start a server listening on port.

ServerSocket server = new ServerSocket(port);
Socket socket = server.accept();

When a client connects the blocking accept() call will return a Socket which is used for communicating with the client.

Writing command output to client

Writing the output from the command to the client means writing the command input stream to the socket output stream. We create an OutputStream from the socket:

// Write output to client
OutputStream output = socket.getOutputStream();

Then write the command output to the socket output stream. Modifying the loop above when reading command output:

String line = "";
while ((line = reader.readLine()) != null) {
    System.out.println(line);
    line += "\n";
    output.write(line.getBytes());
}

Now the simple case with a one-off command is complete. The application accepts a connection, sends the command output to the client and exits. However, if we run an interactive command, like for example bash there is yet no way to write input to the running command. That’s the next task to add.

Writing client input to an executing command

With the simple case done, its time to also read input from the connected client and write to the command output stream. Get the input stream from the socket (i.e. this is where the client command will be received)

// Read from client
InputStream input = socket.getInputStream();

Create a writer for the command stream

OutputStream commandStream = command.getOutputStream();
BufferedWriter writer = new BufferedWriter(
                             new OutputStreamWriter(commandStream));

Then if the command is still running read from the client input stream and write to the command ouput stream.

byte[] buffer = new byte[1024];
while (command.isAlive() &&
    ((bytesRead = input.read(buffer)) != -1))
{
    buffer[bytesRead] = '\n';
    if (command.isAlive()) {
        writer.write(new String(buffer));
        writer.flush();
    } else {
        break;
    }
}

The way this loop is constructed the loop will not exit until the command exits. Reading the input stream is a blocking call, hence the other loop writing the command output to the client will never be reached. So this loop is meant to be run in a sepearate thread, in parallel with the loop writing to client output stream. So put the above in a Runnable

new Thread(new Runnable() {
    @Override
    public void run() {
        // read socket and write to command stream
    }
}).start()

Running the Remote Shell

The remote shell can now be run as a one-off command like this:

$ java RemoteShell 3333 uptime

When connecting with a client the output of the command (here uptime) will be sent to the client and the application will exit:

$ netcat localhost 3333
 17:00:29 up 7 days,  9:05,  1 user,  load average: 1.87, 1.51, 1.34

If the remote shell is given the parameter to start bash it is possible for the client to write commands to the bash command i.e. basically run any command. This is the real remote shell.

$ java ExecCommand 3333 bash

On the client, note that there will be no prompt. Just type the command and see its output.

$ netcat localhost 3333
date
Thu Jan 16 17:14:09 CET 2020
uptime
 17:14:12 up 7 days,  9:19,  1 user,  load average: 1.76, 1.50, 1.45
ls -l /media	
total 4
drwxr-x---+ 2 root root 4096 May  2  2018 bjorn

GitHub source code

The code for this small application can be found here. Enjoy!

Hjortsberg's notes

Building Nginx modules using CMake

Introduction

Determine Nginx build options

Build a dynamic module

Load the dynamic module

Build a static module

References

Set AWS Lambda trigger on an existing S3 bucket in CloudFormation template

Introduction

Bucket notification configuration using Custom Resource

CloudFormation Template

The Custom Resource

Custom Resource handler

Basic idea

Handle non-existing lambda trigger destination

Permissions

Summary

Resources

Introduction to AWS CloudFormation Custom Resource

Introduction

CloudFormation Custom Resource

Resource handler Lambda

Request Type

Error handling

Return value from a Custom Resource

Permissions

Problems and Caveats

Summary

Resources

Hide the Android navigation bar using WindowInsetsController

Introduction

Example

Fullscreen mode

Making your own transient navigation bar

Translucent Navigation bar

Resources

Setup TCP and UDP forwarding to Android Emulator

Introduction

Forward TCP traffic to Android Emulator

Forward UDP traffic to Android Emulator

Resource

Setting up software RAID in linux

Introduction

Installing Software RAID manangement tool

Configuring RAID array

Use the device

Checking the raid array progress

Testing the RAID array

References

Support Google AdMob Ads in Godot Android games

Introduction

Google AdMob

Build export templates with support for third party AdMob module

Getting Godot and the AdMob module

Building

Install Dependencies

Build Godot Libraries for Android

Update Android files

Upgrade AdMob SDK

Build Android export templates

Troubleshooting build problems

Adding Google Ads to the Android game

Showing Ads in Godot

Exporting the game

Example

References

Create Android Game Applications from Godot

Introduction

Setting up Godot for creating native application

Installing export templates

Setting up Android export template

Creating Android applications with Godot

For installation on local device

For publishing on Google Play Store

Tweaking Android details

Linux, Windows and OSX

References

Installing an Android application (.apk) file

Introduction