Screeshot (driver.takeScreenshot()) is flacky

[Yeldar_Kossynbay]

1. Summary

We had try to implement a basic screenshoot testing and found driver.takeSnapshot() it unstable and does not work reliably.

When we run do screenshot, here comes 2 issues:

• screenshoot has an offset
• screenshoot resets the render.

there will be screeshot in the end of the article.

App Name: Joyn
Bug Severity
Select one that applies

Impacts operation of app
Blocks current development
Improvement suggestion
Issue with documentation (If selected, please share the doc link and describe the issue)
Other

2. Steps to reproduce:

There are 3 phases:

1. Appium Installation
2. Running the app
3. Run test script

Phase 1. installation Appium:

Have @amzn:registry in your .npmrc

@amzn:registry=https://k-artifactory-external.labcollab.net/artifactory/api/npm/kepler-npm-prod-local/
//k-artifactory-external.labcollab.net/artifactory/api/npm/kepler-npm-prod-local/:_authToken=${KEPLER_TOKEN}
always-auth=true

Then install things

npm install -g appium@2.2.2 @appium/types ts-node
appium driver install --source=npm @amzn/appium-kepler-driver@3.19.0

Phase 2. Run the app and the appium server.

Launch simulator

kepler device simulator start

kepler device start-port-forwarding --device Simulator -p 8081 --forward false
kepler device start-port-forwarding --device Simulator -p 8097 --forward false
kepler device start-port-forwarding --device Simulator -p 8888 --forward false

When simulator is launched install app.

kepler device install-app -p ./build/x86_64-debug/fireflies_x86_64.vpkg -d Simulator

or

kepler device install-app -p ./build/armv7-debug/fireflies_armv7.vpkg -d Simulator

Now need to run

kepler exec vda shell touch /tmp/automation-toolkit.enable

Then launch the app

kepler device launch-app -a de.prosiebensat1digital.seventv.main -d Simulator

in the app open the first card from the home screen.

Then launch the appium server

appium

Phase 3. Run the test.

I have implemented a very simple test.

import fs from 'fs';
import path from 'path';

import { remote } from 'webdriverio';

const appId = 'de.prosiebensat1digital.seventv';

const capabilities = {
  platformName: 'Kepler',
  'appium:automationName': 'automation-toolkit/JSON-RPC',
  'kepler:device': 'vda://emulator-5554',
  'appium:appURL': `${appId}.main`,
} as const;

const remoteConnectionOpts = {
  hostname: 'localhost',
  port: 4723,
  logLevel: 'info',
  capabilities,
} as const;

const compareScreenshots = (
  screenshotFilename: string,
  screenshotBlob: string
) => {
  const screenshotsDir = path.join(__dirname, 'screenshots');

  if (!fs.existsSync(screenshotsDir)) {
    fs.mkdirSync(screenshotsDir, { recursive: true });
  }

  const screenshotPath = path.join(screenshotsDir, `${screenshotFilename}.png`);

  if (!fs.existsSync(screenshotPath)) {
    // Save screenshot if it doesn't exist
    fs.writeFileSync(screenshotPath, screenshotBlob, 'base64');
    console.log(`✅ Baseline screenshot saved: ${screenshotPath}`);
    return true;
  } else {
    // Compare with existing screenshot
    const baselineScreenshot = fs.readFileSync(screenshotPath, 'base64');

    if (baselineScreenshot === screenshotBlob) {
      console.log(`✅ Screenshots match: ${screenshotPath}`);
      return true;
    } else {
      console.error(`❌ Screenshots do not match: ${screenshotPath}`);

      // Save the failed screenshot for debugging
      const failedPath = screenshotPath.replace('.png', '_failed.png');
      fs.writeFileSync(failedPath, screenshotBlob, 'base64');
      console.log(`📸 Failed screenshot saved: ${failedPath}`);

      return false;
    }
  }
};

const test = async () => {
  console.info('[Test] Screenshot example test - starting');

  const driver = await remote(remoteConnectionOpts);

  try {
    const screenshot = await driver.takeScreenshot(); // <<<<< HERE IS THE SCREENSHOT
    const isMatch = compareScreenshots(screenshot, 'test-test');

    if (!isMatch) {
      throw new Error('Screenshot comparison failed!');
    }

    console.info('[Test] complete');
  } finally {
    await driver.deleteSession();
  }
};

test().catch(console.error);

To tun this test

ts-node ./screenshoot.ts

3. Observed Behavior

Run the test once - There will be created the folder screenshots and inside will be placed a screenshot image file.

Run the test seconds time - The test will fail, because the screeshot are not matching.

1. There is a slight shift of the screenshot. At first the image seems identical, but at closed look you would see that the seconds image has the whole image a bit moved up and left
2. Second screenshot does not show button/card of our UI focused.
3. Sometimes loaded images is reset

Run the test third time - The test is success, because for some unknown reason, the screenshot is matching again.

Run the test 4 time - It would again fail. and this time the screenshot will be absolutely as at the seconds attempt.

4. Expected Behavior

The taken screenshot are identical always, regardless of how many times the screenshots are taken.

Screehshoots:

Attempt #1

On this screenshoot I have catched the case #3 when the loaded and shown image is reset.

Look in the bottom of the image, on the second screenshot the seconds card lsot its’ image

Image #1: intial screenshot

image #2: failed comparison with the first screenshot

Attempt #2:

image #1: initial screenshot

image #2: failed comparison

NOTE:

From this attempt: I have observed a strange behaviour. The screenshot bug is shown on the second screenshot - the button was flipped into the loading state.

The loading state of the button, is actually was a previous render, which was replaced and rerendered into S1 F1 ansehen when the data loading was resolved.

NOTE:

I did not touch the simulator between test running.

I’ve run test only when the page was fully loaded.

I’ve run the both screenshots in a sequence in a row.

6. Environment

Kepler Version: 0.20.3443

• App State: Running & Under testing
• OS information:

kepler device info (this is simulator)

{
  "idme": "A24B7QXDATJDR3",
  "os": "OS",
  "hostname": "amazon-2d41ad3bbb753362",
  "architecture": "aarch64",
  "profile": "tv",
  "product": "vvrp-tv-arm64",
  "buildDescription": "OS 1.1 (VegaMainlineTvIntegration/10164910)",
  "simulated": true,
  "inDeveloperMode": false
}

[Amz_Rsk]

Hi @Yeldar_Kossynbay ,

Our team will check this and get back.

Thanks,
Rohit

[Carlos_Salas]

Hi @Yeldar_Kossynbay,

Thank you for your report. Upon looking at the evidence provided I see that the screenshots are not 100% equal. Image file comparison may not be suitable for your use case if it’s not possible to guarantee that the expected UI is deterministic, for example in your examples, there’s a control still loading and on the first one, there’s an animation or dynamic content which is displayed at the bottom.

Have you looked into other techniques to compare images? If the stable portion of the UI that needs comparing is known, cropping/targeting that image portion would be ideal (i.e. Visual QA). Additionally, your test doesn’t seem to be ensuring the app has reached the desired state to take the screenshot.

Lastly, I suggest implementing an improved Image comparison algorithm. Comparing two PNG files using their Base64 representations is not deterministic as these images are not bitmaps and there may be variations on the file even with small variations not noticeable to the human eye.

Thanks,

Carlos

[Wenguang_Hu]

according to the image you have shared,

I marked the different between images in green rectangle .

[Siva_Prakash]

Hi @Yeldar_Kossynbay,

Were you able to try the suggestions shared above?

Regards,
Siva

[Yeldar_Kossynbay]

Dear @Carlos_Salas

thank you with feedback.

My apologies if I didn’t make it clear, the report is not about how two images compared.

The issue is that calling driver.takeScreenshot() consistently produces different images and may affect the app’s UI rendering.

for example in your examples, there’s a control still loading and on the first one, there’s an animation or dynamic content which is displayed at the bottom.

I confirm that I took screenshots only after the UI of the screen was fully loaded, and no user actions were performed between the captures.

As I mentioned above, my observation is that the driver.takeScreenshot() reverts the UI to the previous rerender.

To illustrate:

Let’s say we have a screen, with 2 buttons. Both buttons are made of something like this:

const Button = () => {
const [isFocused, setState] = useState(false)
....
return (<View style={{backgroundColor: isFocused ? "blue": "green"}}>...</View>)}

1. In the begining, the first button is focused.
2. Then user press RIGHT. it moves focus to the second button
3. make a screenshoot driver.takeScreenshot()

At this moment you may find that the app got weird, it will show that the first button is focused again.

4. make another screenshoot driver.takeScreenshot()

After the second screenshoot the second button will be focused again.

In summary:

On one screenshot one button will be focused, on another screenshot - another button. And for some reason the driver.takeScreenshot() may revert the UI to the previous rerender.

So I am wondering if you are seing this behaviour.

Looking forward hearing from you

With king regards

Yeldar

[Wenguang_Hu]

Hello @Yeldar_Kossynbay
Sorry for the late reply,
We are looking into it.

When do the testing,
do you use the command from appium to press RIGHT?
or you did it on remote controller?

[Wenguang_Hu]

Hello, @Yeldar_Kossynbay
This is a know issue from Virtual Device.

Here is a workaround

you can use command line on simulator’s command line,

to take the screenshot, and take it back

adb shell

cd /data

gwsi-tool-screenshooter my.png

exit

adb pull /data/my.png