Implement Flex Mode into a Unity game
Objective
Learn how to implement Flex mode into a Unity game using Android Jetpack WindowManager and Unity's Java Native Interface (JNI) Wrapper.
Overview
The flexible hinge and glass display on Galaxy foldable devices, such as the Galaxy Z Fold4 and Galaxy Z Flip4, let the phone remains propped open while you use apps. When the phone is partially folded, it will go into Flex mode.
Apps will reorient to fit the screen, letting you watch videos or play games without holding the phone. For example, you can set the device on a flat surface, like on a table, and use the bottom half of the screen to navigate. Unfold the phone to use the apps in full screen mode, and partially fold it again to return to Flex mode. To provide users with a convenient and versatile foldable experience, developers need to optimize their apps to meet the Flex mode standard.
Set up your environment
You will need the following:
-
Unity Hub with Unity 2022.3.5f1 or later (must have Android Build Support)
-
Visual Studio or any source code editor
-
Galaxy Z Fold2 or newer
-
Remote Test Lab (if physical device is not available)
Requirements:
- Samsung account
- Java Runtime Environment (JRE) 7 or later with Java Web Start
- Internet environment where port 2600 is available
Sample Code
Here is a sample project for you to start coding in this Code Lab. Download it and start your learning experience!
Flex Mode on Unity Sample Code (1.17 GB)
Start your project
After downloading the sample project files, follow the steps below to open your project:
- Launch the Unity Hub.
- Click Projects > Open.
- Locate the unzipped project folder and click Open to add the project to the Hub and open in the Editor.
NoteThe sample project was created in Unity 2022.3.5f1. If you prefer using a different Unity version, click Choose another Editor version when prompted and select a higher version of Unity.
To ensure that the project runs smoothly on the Android platform, configure the Player Settings as follows:
- Go to File > Build Settings.
- Under Platform, choose Android and click Switch Platform. Wait until this action finishes importing necessary assets and compiling scripts.
- Then, click Player Settings to open the Project Settings window.
- Go to Player > Other Settings and scroll down to see Target API Level. Set it to API level 33 as any less than this will result in a dependency error regarding an
LStar
variable.
- You can set the Minimum API Level on lower levels without any problem.
- Next, in the Resolution and Presentation settings, enable Resizable Window. It is also recommended that Render outside safe area is enabled to prevent black bars on the edges of the screen.
- Lastly, enable the Custom Main Manifest, Custom Main Gradle Template, and Custom Gradle Properties Template in the Publishing Settings.
- After closing the Project Settings window, check for the new folder structure created within your Assets in the Project window. The newly created Android folder contains
AndroidManifest.xml
, gradleTemplate.properties
, and mainTemplate.gradle
files.
Import the FoldableHelper and add dependencies
FoldableHelper is a Java file that you can use in different projects. It provides an interface to the Android Jetpack WindowManager library, enabling application developers to support new device form factors and multi-window environments.
Before proceeding, read How to Use Jetpack WindowManager in Android Game Dev and learn the details of how FoldableHelper
uses WindowManager
library to retrieve information about the folded state of the device (FLAT
for Normal mode and HALF-OPENED
for Flex mode), window size, and orientation of the fold on the screen.
Download the FoldableHelper.java
file here:
FoldableHelper.java (6.22 KB)
To import the FoldableHelper.java
file and add dependencies to the project, follow the steps below:
- In Assets > Plugins > Android, right-click and select Import New Asset.
- Locate and choose the
FoldableHelper.java
file, then click Import.
-
Next, open the gradleTemplate.properties
file to any source code editor like Visual Studio and add the following lines below the **ADDITIONAL_PROPERTIES**
marker.
android.useAndroidX = true
android.enableJetifier = true
useAndroidX
sets the project to use the appropriate AndroidX libraries instead of support libraries.
enableJetifier
automatically migrates existing third-party libraries to use AndroidX by rewriting their binaries.
-
Lastly, open the mainTemplate.gradle
file and add the dependencies for the artifacts needed for the project.
**APPLY_PLUGINS**
dependencies {
implementation fileTree(dir: 'libs', include: ['*.jar'])
implementation "androidx.appcompat:appcompat:1.6.1"
implementation "androidx.core:core:1.10.1"
implementation "androidx.core:core-ktx:1.10.1"
implementation "androidx.window:window:1.0.0"
implementation "androidx.window:window-java:1.0.0"
implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.9.0"
**DEPS**}
Create a new PlayerActivity
To implement Flex mode on your applications, you must make necessary changes to the Activity
. Since it is impossible to access and change the original UnityPlayerActivity
, you need to create a new PlayerActivity
that inherits from the original. To do this:
-
Create a new file named FoldablePlayerActivity.java
and import it into the Android folder, same as when you imported the FoldableHelper.java
file.
-
To extend the built-in PlayerActivity
from Unity, write below code in the FoldablePlayerActivity.java
file.
package com.unity3d.player;
import android.os.Bundle;
import com.unity3d.player.UnityPlayerActivity;
import com.samsung.android.gamedev.foldable.FoldableHelper;
import com.samsung.android.gamedev.foldable.FoldableHelper.WindowInfoLayoutListener;
import android.util.Log;
public class FoldablePlayerActivity extends UnityPlayerActivity
{
@Override protected void onCreate(Bundle savedInstanceState)
{
super.onCreate(savedInstanceState);
FoldableHelper.init(this);
}
@Override protected void onStart()
{
super.onStart();
FoldableHelper.start(this);
}
@Override protected void onStop()
{
super.onStop();
FoldableHelper.stop();
}
@Override protected void onRestart()
{
super.onRestart();
FoldableHelper.init(this);
}
public void attachUnityListener(WindowInfoLayoutListener listener){
FoldableHelper.attachNativeListener(listener, this);
}
}
onCreate()
calls the FoldableHelper.init()
to ensure that the WindowInfoTracker
and Metrics Calculator gets created as soon as possible.
onStart()
calls the FoldableHelper.start()
since the first WindowLayoutInfo
doesn't get created until onStart()
.
onStop()
calls the FoldableHelper.stop()
to ensure that when the application closes, the listener gets cleaned up.
onRestart()
calls FoldableHelper.init()
when returning to the app after switching away. WindowInfoTracker
must be re-initialized; otherwise, Flex Mode will no longer update.
-
After creating the FoldablePlayerActivity
, ensure that the game uses it. Open the AndroidManifest.xml
file and change the Activity
name to the one you've just created.
<activity android:name="com.unity3d.player.FoldablePlayerActivity"
android:theme="@style/UnityThemeSelector">
…
</activity>
Store FoldableLayoutInfo data to FlexProxy
Implement a native listener that receives calls from Java when the device state changes by following these steps:
-
Use the AndroidJavaProxy provided by Unity in its JNI implementation. AndroidJavaProxy
is a class that implements a Java interface, so the next thing you need to do is create an interface in the FoldableHelper.java
file.
public interface WindowInfoLayoutListener {
void onChanged(FoldableLayoutInfo LayoutInfo);
}
-
This interface replaces the temporary native function. Therefore, remove the code below from the FoldableHelper.java
file:
public static native void OnLayoutChanged(FoldableLayoutInfo resultInfo);
-
Then, go to the Assets > Flex_Scripts folder and right-click to create a new C# script called FlexProxy.cs.
-
Inside this script, replace the automatically generated class with the FlexProxy
class inheriting from AndroidJavaProxy
:
public class FlexProxy : AndroidJavaProxy
{
}
-
In FlexProxy
class, define the variables needed to store the data from FoldableLayoutInfo and use enumerators for the folded state, hinge orientation, and occlusion type. For the various bounds, use Unity's RectInt type. Also, use a boolean to store whether the data has been updated or not.
public enum State { UNDEFINED, FLAT, HALF_OPENED };
public enum Orientation { UNDEFINED, HORIZONTAL, VERTICAL };
public enum OcclusionType { UNDEFINED, NONE, FULL };
public State state = State.UNDEFINED;
public Orientation orientation = Orientation.UNDEFINED;
public OcclusionType occlusionType = OcclusionType.UNDEFINED;
public RectInt foldBounds;
public RectInt currentMetrics;
public RectInt maxMetrics;
public bool needsUpdate = false;
-
Next, define what Java class the FlexProxy
is going to implement by using the interface's fully qualified name as below:
public FlexProxy() : base("com.samsung.android.gamedev.foldable.FoldableHelper$WindowInfoLayoutListener") { }
com.samsung.android.gamedev.foldable
is the package name of the FoldableHelper.java
file.
FoldableHelper$WindowInfoLayoutListener
is the class and interface name separated by a $
.
-
After linking the proxy to the Java interface, create a helper method to simplify Java to native conversions.
private RectInt ConvertToRectInt(AndroidJavaObject rect)
{
if(rect != null)
{
var left = rect.Get<int>("left");
var top = rect.Get<int>("top");
var width = rect.Call<int>("width");
var height = rect.Call<int>("height");
return new RectInt(xMin: left, yMin: top, width: width, height: height);
}
else
{
return new RectInt(-1, -1, -1, -1);
}
}
This method takes a Java Rect object and converts it into a Unity C# RectInt
.
-
Now, use this ConvertToRectInt()
function for the onChanged()
function to retrieve the information from the Java Object and store it in the Flex Proxy Class:
public void onChanged(AndroidJavaObject LayoutInfo)
{
foldBounds = ConvertToRectInt(LayoutInfo.Get<AndroidJavaObject>("bounds"));
currentMetrics = ConvertToRectInt(LayoutInfo.Get<AndroidJavaObject>("currentMetrics"));
maxMetrics = ConvertToRectInt(LayoutInfo.Get<AndroidJavaObject>("maxMetrics"));
orientation = (Orientation)(LayoutInfo.Get<int>("hingeOrientation") + 1);
state = (State)(LayoutInfo.Get<int>("state") + 1);
occlusionType = (OcclusionType)(LayoutInfo.Get<int>("occlusionType") + 1);
needsUpdate = true;
}
Implement native Flex mode
This section focuses on creating the Flex mode split-screen effect on the game’s UI.
-
Create a new C# script in the Flex_Scripts folder called FlexModeManager.cs
.
-
After creating the script, define the variables you need for this implementation.
public class FlexModeManager : MonoBehaviour
{
private FlexProxy windowManagerListener;
[SerializeField]
private Camera mainCamera;
[SerializeField]
private Camera skyBoxCamera;
[SerializeField]
private Canvas controlsCanvas;
[SerializeField]
private Canvas healthCanvas;
[SerializeField]
private GameObject flexBG;
[SerializeField]
private GameObject UIBlur;
windowManagerListener
is the callback object which receives the FoldableLayoutInfo
from the FoldableHelper.java
implementation
mainCamera
and skyBoxCamera
are two cameras to modify in this project for creating a seamless Flex mode implementation
controlsCanvas
and healthCanvas
are the two UI elements to manipulate for implementing the Flex mode
flexBG
is a background image to disable in Normal mode and enable in Flex mode to fill the bottom screen
UIBlur
is a background blur element used as part of the tutorial text. It doesn't function properly with Flex mode, so disabling it when in Flex mode makes the UI looks better.
-
Next, in the Start()
method, create a new instance of the FlexProxy
class and attach it to the Unity application's activity via the attachUnityListener
function.
void Start()
{
windowManagerListener = new FlexProxy();
using (AndroidJavaClass javaClass = new AndroidJavaClass("com.unity3d.player.UnityPlayer"))
{
using (AndroidJavaObject activity = javaClass.GetStatic<AndroidJavaObject>("currentActivity"))
{
activity.Call("attachUnityListener", windowManagerListener);
}
}
}
-
In the Update()
method, check if the windowManagerListener
has received any new data on the folded state of the device.
-
If the system needs an update, then call UpdateFlexMode()
.
void Update()
{
if (windowManagerListener.needsUpdate)
{
UpdateFlexMode();
}
}
-
Create the UpdateFlexMode()
method to enable or disable Flex mode.
NoteThis project is set up for landscape mode only. The implementation discussed in this Code Lab only covers setting up flex mode with a horizontal fold in landscape. However, if you were able to follow, it should be simple to set up something similar for different fold orientations on devices such as the Galaxy Z Flip series of devices.
private void UpdateFlexMode()
{
}
-
Check the folded state of the device via the windowManagerListener
. If the device is HALF_OPENED
, implement Flex mode.
private void UpdateFlexMode()
{
if (windowManagerListener.state == FlexProxy.State.HALF_OPENED)
{
}
}
-
To split the UI screen horizontally, set the anchor points of the controlsCanvas
and the healthCanvas
so they are locked at the bottom screen or below the fold. Also, set the viewports of the mainCamera
and skyBoxCamera
to be above the fold - which is the top screen.
-
Next, set the anchors for the flexBG
object and enable it to fill the space behind the UI on the bottom screen.
-
Deactivate the UIBlur
element if it exists. The UI Blur element is only present at Level 1 of the demo game. A check is necessary to ensure the Flex Mode Manager works on the second level.
private void UpdateFlexMode()
{
if (windowManagerListener.state == FlexProxy.State.HALF_OPENED)
{
float lowerScreenAnchorMax = (float)windowManagerListener.foldBounds.yMin / windowManagerListener.currentMetrics.height;
RectTransform controlsCanvasTransform = controlsCanvas.GetComponent<RectTransform>();
RectTransform healthCanvasTransform = healthCanvas.GetComponent<RectTransform>();
RectTransform flexBGTransform = flexBG.GetComponent<RectTransform>();
controlsCanvasTransform.anchorMin = new Vector2(0, 0);
controlsCanvasTransform.anchorMax = new Vector2(1, lowerScreenAnchorMax);
healthCanvasTransform.anchorMin = new Vector2(0, lowerScreenAnchorMax);
healthCanvasTransform.anchorMax = new Vector2(0, lowerScreenAnchorMax);
flexBGTransform.anchorMin = new Vector2(0, 0);
flexBGTransform.anchorMax = new Vector2(1, lowerScreenAnchorMax);
float upperScreenRectHeight = (float)windowManagerListener.foldBounds.yMax / windowManagerListener.currentMetrics.height;
mainCamera.rect = new Rect(0, upperScreenRectHeight, 1, upperScreenRectHeight);
skyBoxCamera.rect = new Rect(0, upperScreenRectHeight, 1, upperScreenRectHeight);
flexBG.SetActive(true);
if(UIBlur != null)
UIBlur.SetActive(false);
}
}
-
Return the UI to full screen when the device is no longer in Flex mode by:
- disabling
flexBG
;
- enabling
UIBlur
when it exists; and
- setting all the anchor points and viewports back to their original values.
-
And finally, inform the windowManagerListener
that it doesn't need an update.
else
{
RectTransform controlsCanvasTransform = controlsCanvas.GetComponent<RectTransform>();
RectTransform healthCanvasTransform = healthCanvas.GetComponent<RectTransform>();
controlsCanvasTransform.anchorMin = new Vector2(0, 0);
controlsCanvasTransform.anchorMax = new Vector2(1, 1);
healthCanvasTransform.anchorMin = new Vector2(0, 1);
healthCanvasTransform.anchorMax = new Vector2(0, 1);
mainCamera.rect = new Rect(0, 0, 1, 1);
skyBoxCamera.rect = new Rect(0, 0, 1, 1);
flexBG.SetActive(false);
if (UIBlur != null)
UIBlur.SetActive(true);
}
windowManagerListener.needsUpdate = false;
Set up the scenes for Flex Mode
- Go back to the Unity Editor. In Assets > 3DGameKit > Scenes > Gameplay, double-click on the Level 1 scene to open it.
- Right-click in the Hierarchy window and select Create Empty.
- Name the new GameObject as FlexManager or a similar name to reflect its purpose.
- Select the
FlexManager
object and click the Add Component button in the Inspector window. Type in FlexModeManager
, and select the script when it shows up.
- Select the relevant objects for each camera, canvas and game object as below:
- Do the same for Level 2 but leave the UI Blur empty.
Build and run the app
Go to File > Build Settings. Click Build at the bottom of the window to build the APK.
After building the APK, run the game app on a foldable Galaxy device and see how the UI switches from Normal to Flex mode. If you don’t have any physical device, you can also test it on a Remote Test Lab device.
TipWatch this tutorial video and know how to easily test your app via Remote Test Lab.
You're done!
Congratulations! You have successfully achieved the goal of this Code Lab. Now, you can implement Flex mode in your Unity game app by yourself!
To learn more, visit:
www.developer.samsung.com/galaxy-z
www.developer.samsung.com/galaxy-gamedev