Automated testing for multiplayer Game AI Automated testing for - - PowerPoint PPT Presentation

▶
automated testing for multiplayer game ai automated
SMART_READER_LITE
LIVE PREVIEW

Automated testing for multiplayer Game AI Automated testing for - - PowerPoint PPT Presentation

Oct 03, 2023 222 likes •790 views

Automated testing for multiplayer Game AI Automated testing for multiplayer Game AI in Sea of Thieves in Sea of Thieves Robert Masella Rare - Microsoft Studios About Me About Me Senior Gameplay Engineer Worked on: Banjo Kazooie:

slide-1
SLIDE 1

Automated testing for multiplayer Game AI in Sea of Thieves Automated testing for multiplayer Game AI in Sea of Thieves

Robert Masella

Rare - Microsoft Studios

slide-2
SLIDE 2

About Me About Me

  • Senior Gameplay Engineer
  • Worked on:
  • Banjo Kazooie: Nuts and Bolts
  • Kinect Sports games
  • Sea of Thieves
  • Two years on AI
slide-3
SLIDE 3

Rare Rare

  • Microsoft first party studio
  • 30+ years of games
slide-4
SLIDE 4

Sea of Thieves Sea of Thieves

slide-5
SLIDE 5

Sea of Thieves Sea of Thieves

slide-6
SLIDE 6

What is Sea of Thieves? What is Sea of Thieves?

  • ‘Shared World Adventure Game’
  • Two main AI threats:
  • Skeletons on islands
  • Sharks in the sea
  • Other AI types being worked on
slide-7
SLIDE 7

Sea of Thieves Tech Sea of Thieves Tech

  • Uses Unreal Engine 4
  • Dedicated servers
  • AI processing runs on server
  • AI uses UE4 behaviour trees
  • With many extended classes
slide-8
SLIDE 8

This Talk This Talk

  • How we shipped weekly
  • AI behaving correctly on every build
  • Using automated testing and continuous delivery
slide-9
SLIDE 9

Why use automated tests? Why use automated tests?

  • Game as a service means constant build confidence necessary
  • Reduce manual testing
  • Reduce crunch!
slide-10
SLIDE 10

Testing AI Testing AI

  • Automated testing not widely used in game development
  • AI Unique challenges for testing AI
  • Multiplayer
slide-11
SLIDE 11

Imagine this scenario Imagine this scenario

  • Designer changes a number in AI perception asset
  • Causes AI to forget players leaving Line of Sight correctly
  • Not spotted by manual test
  • Bug released to players
slide-12
SLIDE 12

Consequences Consequences

  • Weeks later…
  • Players start to notice in build
  • More engineer time to track down
  • More testing time to verify fix
  • Every chance issue could reoccur
slide-13
SLIDE 13

How to avoid How to avoid

  • Add an automated test
  • Run it regularly
  • Now bug would be caught before release
  • Ideally before it enters build
slide-14
SLIDE 14

Testing at Rare Testing at Rare

  • Require testing with every check in
  • Sea of Thieves now has 10,000 tests
  • Test run remotely using TeamCity
slide-15
SLIDE 15

Automated Testing Organisation Automated Testing Organisation

  • Run core set of tests before every check in
  • Run longer running tests periodically
  • Run all tests multiple times:
  • Different platforms
  • Different configurations (e.g. shipping, debug)
  • Different conditions (e.g. high latency, high packet variance)
slide-16
SLIDE 16

Release Process Release Process

  • Fix gets to players in 30 hrs:

0 hrs 0 min Fix implementation completed 0 hrs 30 min Fix verified on our build farm 0 hrs 30 min Fix submitted 3 hrs 0 min Build built, deployed to test lab and testing begins 5 hrs 0 min Entire build verified 29hrs 0 min Build passed through certification, ready to be published to retail. 30hrs 0 min Build published to retail and available to players

slide-17
SLIDE 17

Test Types Test Types

  • Unit
  • Integration
  • Behaviour Tree
  • Multiplayer Integration
slide-18
SLIDE 18

Test Types Test Types

  • Favour unit tests over integration tests
  • 90% of tests are unit tests
  • Unit tests don’t cover interaction between big systems
  • And rarely test production assets
  • Ideal to have coverage by all test types
slide-19
SLIDE 19

Testing Standards Testing Standards

  • Names follow Given_When_Then pattern
  • E.g. AIWithHearingSense_ReceivesNoiseEvent_AIAcknowledgesNoise
  • Each test tests one thing
  • Make sure you’ve seen the test fail as well as pass
  • Minimise boilerplate
slide-20
SLIDE 20

Running Tests Running Tests

  • Run tests in Unreal Editor Automation window
slide-21
SLIDE 21

Unit Tests Unit Tests

  • Run without creating or ticking world
  • Unit tests can have test fixture for boilerplate code

#define IMPLEMENT_AIENTITY_TEST( TestName ) IMPLEMENT_UNIT_TEST( TestName, "AIEntity", AIEntityTestFixture ) IMPLEMENT_AIENTITY_TEST( UpdateTarget_OneEnemyEntitySeen_TargetSetToEntity ) { auto* AIEntity = SpawnTestAIEntity(); auto* EnemyEntity = SpawnEnemyEntity(); AIEntity->AddSeenEntity( EnemyEntity ); AIEntity->UpdateTarget(); TestEqual( AIEntity->GetTargetEntity(), EnemyEntity ); }

slide-22
SLIDE 22

Integration Tests Integration Tests

  • Mostly used Unreal Blueprint system:
  • https://docs.unrealengine.com/latest/INT/Engine/Blueprints/
slide-23
SLIDE 23

Integration Tests Integration Tests

  • Levels with limited game scenario
  • Most look for success criteria, or time out and fail
slide-24
SLIDE 24

Line of Sight integration test Line of Sight integration test

  • SkeletonAI_WhenLosesLineOfSightToTarget_MovesToPositionToRegainLineOfSight
slide-25
SLIDE 25

Line of Sight integration test Line of Sight integration test

slide-26
SLIDE 26

Line of Sight integration test Line of Sight integration test

  • Failing version
slide-27
SLIDE 27

Line of Sight integration test Line of Sight integration test

slide-28
SLIDE 28

Line of Sight integration test Line of Sight integration test

  • Failing version
  • Assertion: Test Timed Out
slide-29
SLIDE 29

Line of Sight integration test Line of Sight integration test

  • Passing version
slide-30
SLIDE 30

Line of Sight integration test Line of Sight integration test

slide-31
SLIDE 31

Line of Sight integration test Line of Sight integration test

  • Passing version
slide-32
SLIDE 32

Behaviour Tree Testing Behaviour Tree Testing

  • In Unreal, behaviour trees are assets
  • Tested with integration tests.
  • Also required testing for new node types
  • Preferred testing with unit tests
slide-33
SLIDE 33

Example Behaviour Tree Node Example Behaviour Tree Node

  • Node that triggers an input
slide-34
SLIDE 34

Behaviour Tree Node Testing Behaviour Tree Node Testing

  • Add node to a minimal behaviour tree created in code
  • Created environment required for coded behaviour tree in fixture
  • Also created helper functions for adding nodes

virtual void OnBeforeTest() override { BehaviorTree = CreateTreeRootWithSequence(); } UBTTask_TriggerInput* CreateTriggerInputTaskNodeAttachedToNode( UBTCompositeNode* ParentNode, UNotificationInputId NotificationId ) { auto* TestTask = NewObject< UBTTask_TriggerInput >(); TestTask->NotificationId = NotificationId; ParentNode->AddChild( TestTask ); return TestTask; }

slide-35
SLIDE 35

Node Testing Node Testing

  • Test that checks that input is triggered and of correct type

IMPLEMENT_UNIT_TEST_BEHAVIOUR_TREE_NODE( TriggerInputNode_NodeIsRun_CallsCorrectInputNotification ) { bool TestNotificationInput1Called = false; TestEntity->HandleNotificationCallback = [&]( UNotificationInputId NotificationInputId ) { if ( NotificationInputId == UTestNotificationInputId ) { TestNotificationInput1Called = true; } }; NotificationNode = CreateTriggerInputTaskNodeAttachedToNode( RootNode, UTestNotificationInputId ); RunTreeWithInitialTick(); TestTrue( TestNotificationInput1Called ); }

slide-36
SLIDE 36

Node Testing Node Testing

  • Test that checks that

input is triggered and of correct type

slide-37
SLIDE 37

Example Behaviour Tree Node II Example Behaviour Tree Node II

  • Decorator node that compares entity’s health
slide-38
SLIDE 38

Node Testing II Node Testing II

  • Test that checks that decorator fails if health is lower than expected

IMPLEMENT_UNIT_TEST_BEHAVIOUR_TREE_NODE( CompareCurrentHealth_CompareGreaterThanValueHealthIsLess_DecoratorNodeFails ) { bool TaskWithDecoratorExecuted = false; bool LowerPriorityTaskExecuted = false; auto* Decorator = CreateCompareCurrentHealthDecorator( EFloatValueComparisonType::GreaterThan, 0.1f ); AddTestExecutionChildNodeWithDecorator( RootNode, [&]() { TaskWithDecoratorExecuted = true; }, Decorator ); AddTestExecutionChildNode( RootNode, [&]() { LowerPriorityTaskExecuted = true; } ); AIEntity->SetHealth( 0.05f ); RunTreeWithInitialTick(); TestFalse( TaskWithDecoratorExecuted ); TestTrue( LowerPriorityTaskExecuted ); }

slide-39
SLIDE 39

Node Testing II Node Testing II

  • Test that checks that

decorator fails if health is lower than expected

slide-40
SLIDE 40

Multiplayer Integration Testing Multiplayer Integration Testing

  • Most AI integration tests require server only
  • Sometimes AI tests require client
  • Integration test framework supports tests over network
slide-41
SLIDE 41

Multiplayer Integration Testing Multiplayer Integration Testing

  • Use Yield Nodes to pass execution between server and clients
slide-42
SLIDE 42

‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test

  • SkeletonAI_Dies_DoesNotMoveDuringDeathOnClient_MP1
slide-43
SLIDE 43

‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test

slide-44
SLIDE 44

‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test

  • Failing version
slide-45
SLIDE 45

‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test

slide-46
SLIDE 46

‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test

  • Failing version
  • Assertion failed: ‘check velocity is low in dead state’
slide-47
SLIDE 47

‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test

  • Passing version
slide-48
SLIDE 48

‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test

slide-49
SLIDE 49

‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test

  • Passing version
slide-50
SLIDE 50

Testing drawbacks Testing drawbacks

  • Time to create tests
  • Tests on an evolving game
  • Intermittent integration tests
slide-51
SLIDE 51

Testing benefits Testing benefits

  • Shipped over 100 times in 2 years
  • Very low bug count
  • Improved code
  • Manual testers focus on other things
  • No crunch for 4 years!
slide-52
SLIDE 52

Conclusion - AI In Action Conclusion - AI In Action

  • Now lets see it altogether
slide-53
SLIDE 53

AI In Action AI In Action

slide-54
SLIDE 54

Rare Are Hiring Rare Are Hiring

www.rare.co.uk/careers

Particularly looking for gameplay and AI engineers!

slide-55
SLIDE 55

Questions? Questions?